text:
# MagicMirror² 모듈 개발 문서

이 문서에는 당신만의 MagicMirror² 모듈을 만드는 방법에 대해서 설명되어 있습니다.

## 모듈의 구성

모든 모듈은 `modules` 폴더에 저장되어있으며, 기본 모듈은 `modules/default` 폴더에 함께 저장되어있습니다. 또한, 당신이 만든 모듈은 `modules` 폴더 아래의 또 다른 하위 폴더에 저장되어야합니다.
당신이 `modules` 폴더 속에 만든 모든 파일과 폴더는 새로운 버전의 MagicMirror²를 받을 때 손상될 수 없게, git으로부터 무시된다는 것을 알아두시길 바랍니다.

하나의 폴더에는 하나의, 혹은 그보다 많은 양의 모듈이 존재할 수 있습니다. 단, 각 모듈의 이름은 중복될 수 없습니다.  비슷한(혹은 같은) 이름의 모듈이 서로 다른 폴더에 존재한다고 하더라도, 해당 모듈을 동시에 로딩할 수 없다는 것을 명심하십시오.

### 파일 구성
- **modulename/modulename.js** - 당신이 만든 모듈의 core 모듈 스크립트입니다.
- **modulename/node_helper.js** - node 스크립트에 의하여 불려질 helper 스크립트입니다. helper 스크립트와 core 스크립트는 별도로 통합된 소켓 시스템에 의해 서로에게 정보를 전달할 수 있습니다. 해당 파일은 필수가 아닙니다(optional).
- **modulename/public** - 해당 폴더에 있는 모든 파일은 브라우저 내에서 `/modulename/filename.ext`과 같이 접근 가능합니다.
- **modulename/anyfileorfolder** - 해당 폴더에 있는 모든 파일 및 폴더는 core 모듈 스크립트에서 접근 가능합니다. 예컨대, *modulename/css/modulename.css*는 추가적인 모듈 스타일을 관리하기 위한 좋은 경로가 될 수 있습니다.

## Core 모듈 파일: modulename.js
모듈의 내용이 정의되는 필수 스크립트 파일입니다. 가장 간단하게, 다음과 같이 작성될 수 있습니다:
````javascript
Module.register("modulename",{});
````

물론 위의 모듈은 아무런 추가적인 작업을 할 수 없기 때문에, 간단한 모듈 중 하나인 **helloworld**의 core 모듈 스크립트를 살펴보며 기본적인 설명을 덧붙이도록 하겠습니다:
````javascript
//helloworld.js:

Module.register("helloworld",{
	// Default module config.
	defaults: {
		text: "Hello World!"
	},

	// Override dom generator.
	getDom: function() {
		var wrapper = document.createElement("div");
		wrapper.innerHTML = this.config.text;
		return wrapper;
	}
});
````

위 스크립트에서 `Module.register()`는 두 개의 인수를 받고 있습니다: 모듈의 이름과 모듈의 세부적인 설정에 관한 내용이 그것입니다.

### 참조 가능한 모듈 객체의 속성 값
모듈이 초기화되고 나면, 기본적인 모듈 객체의 속성의 값을 참조할 수 있습니다.

####`this.name`
**String**

모듈의 이름을 나타냅니다.

####`this.identifier`
**String**

해당 모듈 객체의 고유번호(unique identifier)를 나타냅니다.

####`this.hidden`
**Boolean**

해당 모듈의 모습이 숨겨졌는지(사라졌는지)에 대한 Boolean값을 나타냅니다.

####`this.config`
**Boolean**

사용자의 config.js 파일에 작성해둔 모듈의 속성 값입니다.
사용자의 config을 통한 추가적인 설정이 없었다면, 모듈이 가질 수 있는 기본 속성을 갖습니다.

####`this.data`
**Object**

모듈 객체에 대한 metadata를 포함하고 있는 data 객체입니다.
- `data.classes` - 모듈 dom wrapper에 추가된 classes의 정보입니다.
- `data.file` - core 모듈 파일의 이름입니다.
- `data.path` - 해당 모듈이 존재하는 폴더의 경로입니다.
- `data.header` - 모듈에 추가된 header의 정보입니다.
- `data.position` - 모듈 객체가 표시될 위치의 정보입니다.

####`defaults: {}`
당신이 만든 모듈의 기본 속성을 작성할 수 있는 객체입니다. 해당 객체에 작성된 속성 값은 사용자의 config.js의 속성 값과 함께 사용될 것입니다. 모든 모듈 속성 값은 `this.config.propertyName`와 같이 참조될 수 있습니다. 이후 해당 부분에 대해 더 자세히 알려드리도록 하겠습니다.

####'requiresVersion:'

*2.1.0 버전에 추가된 내용입니다.*

모듈을 구동시키기에 필요한 MagicMirror 프레임워크의 최소 버전입니다. 만약 따로 정의되어있지 않다면, 시스템이 자동적으로 모듈의 현재 버전과 MagicMirror의 현재 버전을 비교할 것입니다. 만약 모듈의 현재 버전의 업데이트가 필요하다고 판단되면, MagicMirror는 모듈을 구동하지 않을 것입니다. Node helper 스크립트에서 해당 값을 확실하게 작성하시길 바랍니다.

**주의:** 해당 속성은 MagicMirror 버전 2.1.0에서 소개되었기 때문에, 그보다 오래된 버전에서는 작동하지 않을 것입니다. issue를 작성하기 전에 꼭 참고하시길 바랍니다.

예시:
````javascript
requiresVersion: "2.1.0",
````

### 내부적으로 사용가능한 모듈 함수

####`init()`
This method is called when a module gets instantiated. In most cases you do not need to subclass this method.

####`loaded(callback)`

*Introduced in version: 2.1.1.*

This method is called when a module is loaded. Subsequent modules in the config are not yet loaded. The `callback` function MUST be called when the module is done loading. In most cases you do not need to subclass this method.

**Example:**
````javascript
loaded: function(callback) {
	this.finishLoading();
	Log.log(this.name + ' is loaded!');
	callback();
}
````

####`start()`
This method is called when all modules are loaded an the system is ready to boot up. Keep in mind that the dom object for the module is not yet created. The start method is a perfect place to define any additional module properties:

**Example:**
````javascript
start: function() {
	this.mySpecialProperty = "So much wow!";
	Log.log(this.name + ' is started!');
}
````

####`getScripts()`
**Should return: Array**

The getScripts method is called to request any additional scripts that need to be loaded. This method should therefore return an array with strings. If you want to return a full path to a file in the module folder, use the `this.file('filename.js')` method. In all cases the loader will only load a file once. It even checks if the file is available in the default vendor folder.

**Example:**
````javascript
getScripts: function() {
	return [
		'script.js', // will try to load it from the vendor folder, otherwise it will load is from the module folder.
		'moment.js', // this file is available in the vendor folder, so it doesn't need to be available in the module folder.
		this.file('anotherfile.js'), // this file will be loaded straight from the module folder.
		'https://code.jquery.com/jquery-2.2.3.min.js',  // this file will be loaded from the jquery servers.
	]
}

````
**Note:** If a file can not be loaded, the boot up of the mirror will stall. Therefore it's advised not to use any external urls.

####`getStyles()`
**Should return: Array**

The getStyles method is called to request any additional stylesheets that need to be loaded. This method should therefore return an array with strings. If you want to return a full path to a file in the module folder, use the `this.file('filename.css')` method. In all cases the loader will only load a file once. It even checks if the file is available in the default vendor folder.

**Example:**
````javascript
getStyles: function() {
	return [
		'script.css', // will try to load it from the vendor folder, otherwise it will load is from the module folder.
		'font-awesome.css', // this file is available in the vendor folder, so it doesn't need to be avialable in the module folder.
		this.file('anotherfile.css'), // this file will be loaded straight from the module folder.
		'https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css',  // this file will be loaded from the bootstrapcdn servers.
	]
}

````
**Note:** If a file can not be loaded, the boot up of the mirror will stall. Therefore it's advised not to use any external urls.

####`getTranslations()`
**Should return: Dictionary**

The getTranslations method is called to request translation files that need to be loaded. This method should therefore return a dictionary with the files to load, identified by the country's short name.

If the module does not have any module specific translations, the function can just be omitted or return `false`.

**Example:**
````javascript
getTranslations: function() {
	return {
			en: "translations/en.json",
			de: "translations/de.json"
	}
}

````

####`getDom()`
**Should return:** Dom Object

Whenever the MagicMirror needs to update the information on screen (because it starts, or because your module asked a refresh using `this.updateDom()`), the system calls the getDom method. This method should therefore return a dom object.

**Example:**
````javascript
getDom: function() {
	var wrapper = document.createElement("div");
	wrapper.innerHTML = 'Hello world!';
	return wrapper;
}

````

####`getHeader()`
**Should return:** String

Whenever the MagicMirror needs to update the information on screen (because it starts, or because your module asked a refresh using `this.updateDom()`), the system calls the getHeader method to retrieve the module's header. This method should therefor return a string. If this method is not subclassed, this function will return the user's configured header.

If you want to use the original user's configured header, reference `this.data.header`.

**NOTE:** If the user did not configure a default header, no header will be displayed and thus this method will not be called.

**Example:**
````javascript
getHeader: function() {
	return this.data.header + ' Foo Bar';
}

````

####`notificationReceived(notification, payload, sender)`

That MagicMirror core has the ability to send notifications to modules. Or even better: the modules have the possibility to send notifications to other modules. When this module is called, it has 3 arguments:

- `notification` - String - The notification identifier.
- `payload` - AnyType - The payload of a notification.
- `sender` - Module - The sender of the notification. If this argument is `undefined`, the sender of the notififiction is the core system.

**Example:**
````javascript
notificationReceived: function(notification, payload, sender) {
	if (sender) {
		Log.log(this.name + " received a module notification: " + notification + " from sender: " + sender.name);
	} else {
		Log.log(this.name + " received a system notification: " + notification);
	}
}
````

**Note:** the system sends two notifications when starting up. These notifications could come in handy!

- `ALL_MODULES_STARTED` - All modules are started. You can now send notifications to other modules.
- `DOM_OBJECTS_CREATED` - All dom objects are created. The system is now ready to perform visual changes.

####`socketNotificationReceived: function(notification, payload)`
When using a node_helper, the node helper can send your module notifications. When this module is called, it has 2 arguments:

- `notification` - String - The notification identifier.
- `payload` - AnyType - The payload of a notification.

**Note 1:** When a node helper sends a notification, all modules of that module type receive the same notifications.

**Note 2:** The socket connection is established as soon as the module sends its first message using [sendSocketNotification](thissendsocketnotificationnotification-payload).

**Example:**
````javascript
socketNotificationReceived: function(notification, payload) {
	Log.log(this.name + " received a socket notification: " + notification + " - Payload: " + payload);
},
````

####`suspend()`
When a module is hidden (using the `module.hide()` method), the `suspend()` method will be called. By subclassing this method you can perform tasks like halting the update timers.

####`resume()`
When a module will be shown after it was previously hidden (using the `module.show()` method), the `resume()` method will be called. By subclassing this method you can perform tasks restarting the update timers.

### Module instance methods

Each module instance has some handy methods which can be helpful building your module.

####`this.file(filename)`
***filename* String** - The name of the file you want to create the path for.

**Returns String**

If you want to create a path to a file in your module folder, use the `file()` method. It returns the path to the filename given as the attribute. Is method comes in handy when configuring the [getScripts](#getscripts) and [getStyles](#getstyles) methods.

####`this.updateDom(speed)`
***speed* Number** - Optional. Animation speed in milliseconds.

Whenever your module need to be updated, call the `updateDom(speed)` method. It requests the MagicMirror core to update its dom object. If you define the speed, the content update will be animated, but only if the content will really change.

As an example: the clock modules calls this method every second:

````javascript
...
start: function() {
	var self = this;
	setInterval(function() {
		self.updateDom(); // no speed defined, so it updates instantly.
	}, 1000); //perform every 1000 milliseconds.
},
...
````

####`this.sendNotification(notification, payload)`
***notification* String** - The notification identifier.

***payload* AnyType** - Optional. A notification payload.

If you want to send a notification to all other modules, use the `sendNotification(notification, payload)`. All other modules will receive the message via the [notificationReceived](#notificationreceivednotification-payload-sender) method. In that case, the sender is automatically set to the instance calling the sendNotification method.

**Example:**
````javascript
this.sendNotification('MYMODULE_READY_FOR_ACTION', {foo:bar});
````

####`this.sendSocketNotification(notification, payload)`
***notification* String** - The notification identifier.

***payload* AnyType** - Optional. A notification payload.

If you want to send a notification to the node_helper, use the `sendSocketNotification(notification, payload)`. Only the node_helper of this module will receive the socket notification.

**Example:**
````javascript
this.sendSocketNotification('SET_CONFIG', this.config);
````

####`this.hide(speed, callback, options)`
***speed* Number** - Optional (Required when setting callback or options), The speed of the hide animation in milliseconds.
***callback* Function** - Optional, The callback after the hide animation is finished.
***options* Function** - Optional, Object with additional options for the hide action (see below). (*Introduced in version: 2.1.0.*)

To hide a module, you can call the `hide(speed, callback)` method. You can call the hide method on the module instance itself using `this.hide()`, but of course you can also hide another module using `anOtherModule.hide()`.

Possible configurable options:

- `lockString` - String - When setting lock string, the module can not be shown without passing the correct lockstring. This way (multiple) modules can prevent a module from showing. It's considered best practice to use your modules identifier as the locksString: `this.identifier`. See *visibility locking* below.

**Note 1:** If the hide animation is canceled, for instance because the show method is called before the hide animation was finished, the callback will not be called.

**Note 2:** If the hide animation is hijacked (an other method calls hide on the same module), the callback will not be called.

**Note 3:** If the dom is not yet created, the hide method won't work. Wait for the `DOM_OBJECTS_CREATED` [notification](#notificationreceivednotification-payload-sender).

####`this.show(speed, callback, options)`
***speed* Number** - Optional (Required when setting callback or options), The speed of the show animation in milliseconds.
***callback* Function** - Optional, The callback after the show animation is finished.
***options* Function** - Optional, Object with additional options for the show action (see below). (*Introduced in version: 2.1.0.*)

To show a module, you can call the `show(speed, callback)` method. You can call the show method on the module instance itself using `this.show()`, but of course you can also show another module using `anOtherModule.show()`.

Possible configurable options:

- `lockString` - String - When setting lock string, the module can not be shown without passing the correct lockstring. This way (multiple) modules can prevent a module from showing. See *visibility locking* below.
- `force` - Boolean - When setting the force tag to `true`, the locking mechanism will be overwritten. Use this option with caution. It's considered best practice to let the usage of the force option be use- configurable. See *visibility locking* below.

**Note 1:** If the show animation is canceled, for instance because the hide method is called before the show animation was finished, the callback will not be called.

**Note 2:** If the show animation is hijacked (an other method calls show on the same module), the callback will not be called.

**Note 3:** If the dom is not yet created, the show method won't work. Wait for the `DOM_OBJECTS_CREATED` [notification](#notificationreceivednotification-payload-sender).

####Visibility locking

(*Introduced in version: 2.1.0.*)

Visiblity locking helps the module system to prevent unwanted hide/show actions. The following scenario explains the concept:

**Module B asks module A to hide:**
````javascript
moduleA.hide(0, {lockString: "module_b_identifier"});
````
Module A is now hidden, and has an lock array with the following strings:
````javascript
moduleA.lockStrings == ["module_b_identifier"]
moduleA.hidden == true
````
**Module C asks module A to hide:**
````javascript
moduleA.hide(0, {lockString: "module_c_identifier"});
````
Module A is now hidden, and has an lock array with the following strings:
````javascript
moduleA.lockStrings == ["module_b_identifier", "module_c_identifier"]
moduleA.hidden == true
````
**Module B asks module A to show:**
````javascript
moduleA.show(0, {lockString: "module_b_identifier"});
````
The lockString will be removed from moduleA’s locks array, but since there still is an other lock string available, the module remains hidden:
````javascript
moduleA.lockStrings == ["module_c_identifier"]
moduleA.hidden == true
````
**Module C asks module A to show:**
````javascript
moduleA.show(0, {lockString: "module_c_identifier"});
````
The lockString will be removed from moduleA’s locks array, and since this will result in an empty lock array, the module will be visible:
````javascript
moduleA.lockStrings == []
moduleA.hidden == false
````

**Note:** The locking mechanism can be overwritten by using the force tag:
````javascript
moduleA.show(0, {force: true});
````
This will reset the lockstring array, and will show the module.
````javascript
moduleA.lockStrings == []
moduleA.hidden == false
````

Use this `force` method with caution. See `show()` method for more information.

####`this.translate(identifier)`
***identifier* String** - Identifier of the string that should be translated.

The Magic Mirror contains a convenience wrapper for `l18n`. You can use this to automatically serve different translations for your modules based on the user's `language` configuration.

If no translation is found, a fallback will be used. The fallback sequence is as follows:
- 1. Translation as defined in module translation file of the user's preferred language.
- 2. Translation as defined in core translation file of the user's preferred language.
- 3. Translation as defined in module translation file of the fallback language (the first defined module translation file).
- 4. Translation as defined in core translation file of the fallback language (the first defined core translation file).
- 5. The key (identifier) of the translation.

When adding translations to your module, it's a good idea to see if an apropriate translation is already available in the [core translation files](https://github.com/MichMich/MagicMirror/tree/master/translations). This way, your module can benefit from the existing translations.

**Example:**
````javascript
this.translate("INFO") //Will return a translated string for the identifier INFO
````

**Example json file:**
````javascript
{
  "INFO": "Really important information!"
}
````

**Note:** although comments are officially not supported in JSON files, MagicMirror allows it by stripping the comments before parsing the JSON file. Comments in translation files could help other translators.

## The Node Helper: node_helper.js

The node helper is a Node.js script that is able to do some backend task to support your module. For every module type, only one node helper instance will be created. For example: if your MagicMirror uses two calendar modules, there will be only one calendar node helper instantiated.

**Note:** Because there is only one node helper per module type, there is no default config available within your module. It's your task to send the desired config from your module to your node helper.

In it's most simple form, the node_helper.js file must contain:

````javascript
var NodeHelper = require("node_helper");
module.exports = NodeHelper.create({});
````

Of course, the above helper would not do anything useful. So with the information above, you should be able to make it a bit more sophisticated.

### Available module instance properties

####`this.name`
**String**

The name of the module

####`this.path`
**String**

The path of the module

####`this.expressApp`
**Express App Instance**

This is a link to the express instance. It will allow you to define extra routes.

**Example:**
````javascript
start: function() {
	this.expressApp.get('/foobar', function (req, res) {
		res.send('GET request to /foobar');
	});
}
````

**Note:** By default, a public path to your module's public folder will be created:
````javascript
this.expressApp.use("/" + this.name, express.static(this.path + "/public"));
````

####`this.io`
**Socket IO Instance**

This is a link to the IO instance. It will allow you to do some Socket.IO magic. In most cases you won't need this, since the Node Helper has a few convenience methods to make this simple.

####'requiresVersion:'
*Introduced in version: 2.1.0.*

A string that defines the minimum version of the MagicMirror framework. If it is set, the system compares the required version with the users version. If the version of the user is out of date, it won't run the module.

**Note:** Since this check is introduced in version 2.1.0, this check will not be run in older versions. Keep this in mind if you get issue reports on your module.

Example:
````javascript
requiresVersion: "2.1.0",
````

### Subclassable module methods

####`init()`
This method is called when a node helper gets instantiated. In most cases you do not need to subclass this method.

####`start()`
This method is called when all node helpers are loaded and the system is ready to boot up. The start method is a perfect place to define any additional module properties:

**Example:**
````javascript
start: function() {
	this.mySpecialProperty = "So much wow!";
	Log.log(this.name + ' is started!');
}
````

####`socketNotificationReceived: function(notification, payload)`
With this method, your node helper can receive notifications from your modules. When this method is called, it has 2 arguments:

- `notification` - String - The notification identifier.
- `payload` - AnyType - The payload of a notification.

**Note:** The socket connection is established as soon as the module sends its first message using [sendSocketNotification](thissendsocketnotificationnotification-payload).

**Example:**
````javascript
socketNotificationReceived: function(notification, payload) {
	Log.log(this.name + " received a socket notification: " + notification + " - Payload: " + payload);
},
````

### Module instance methods

Each node helper has some handy methods which can be helpful building your module.

####`this.sendSocketNotification(notification, payload)`
***notification* String** - The notification identifier.

***payload* AnyType** - Optional.  Only the module of your module type will receive the socket notification.

**Note:** Since all instances of your module will receive the notifications, it's your task to make sure the right module responds to your messages.

**Example:**
````javascript
this.sendSocketNotification('SET_CONFIG', this.config);
````

## MagicMirror Helper Methods

The core Magic Mirror object: `MM` has some handy method that will help you in controlling your and other modules. Most of the `MM` methods are available via convenience methods on the Module instance.

### Module selection
The only additional method available for your module, is the feature to retrieve references to other modules. This can be used to hide and show other modules.

####`MM.getModules()`
**Returns Array** - An array with module instances.

To make a selection of all currently loaded module instances, run the `MM.getModules()` method. It will return an array with all currently loaded module instances. The returned array has a lot of filtering methods. See below for more info.

**Note:** This method returns an empty array if not all modules are started yet. Wait for the `ALL_MODULES_STARTED` [notification](#notificationreceivednotification-payload-sender).

#####`.withClass(classnames)`
***classnames* String or Array** - The class names on which you want to filter.
**Returns Array** - An array with module instances.

If you want to make a selection based on one or more class names, use the withClass method on a result of the `MM.getModules()` method. The argument of the `withClass(classname)` method can be an array, or space separated string.

**Examples:**
````javascript
var modules = MM.getModules().withClass('classname');
var modules = MM.getModules().withClass('classname1 classname2');
var modules = MM.getModules().withClass(['classname1','classname2']);
````

#####`.exceptWithClass(classnames)`
***classnames* String or Array** - The class names of the modules you want to remove from the results.
**Returns Array** - An array with module instances.

If you to remove some modules from a selection based on a classname, use the exceptWithClass method on a result of the `MM.getModules()` method. The argument of the `exceptWithClass(classname)` method can be an array, or space separated string.

**Examples:**
````javascript
var modules = MM.getModules().exceptWithClass('classname');
var modules = MM.getModules().exceptWithClass('classname1 classname2');
var modules = MM.getModules().exceptWithClass(['classname1','classname2']);
````

#####`.exceptModule(module)`
***module* Module Object** - The reference to a module you want to remove from the results.
**Returns Array** - An array with module instances.

If you to remove a specific module instance from a selection based on a classname, use the exceptWithClass method on a result of the `MM.getModules()` method. This can be helpful if you want to select all module instances except the instance of your module.

**Examples:**
````javascript
var modules = MM.getModules().exceptModule(this);
````

Of course, you can combine all of the above filters:

**Example:**
````javascript
var modules = MM.getModules().withClass('classname1').exceptwithClass('classname2').exceptModule(aModule);
````

#####`.enumerate(callback)`
***callback* Function(module)** - The callback run on every instance.

If you want to perform an action on all selected modules, you can use the `enumerate` function:

````javascript
MM.getModules().enumerate(function(module) {
    Log.log(module.name);
});
````

**Example:**
To hide all modules except the your module instance, you could write something like:
````javascript
Module.register("modulename",{
	//...
	notificationReceived: function(notification, payload, sender) {
		if (notification === 'DOM_OBJECTS_CREATED') {
			MM.getModules().exceptModule(this).enumerate(function(module) {
				module.hide(1000, function() {
					//Module hidden.
				});
			});
		}
	},
	//...
});
````

## MagicMirror Logger

The Magic Mirror contains a convenience wrapper for logging. Currently, this logger is a simple proxy to the original `console.log` methods. But it might get additional features in the future. The Loggers is currently only available in the core module file (not in the node_helper).

**Examples:**
````javascript
Log.info('error');
Log.log('log');
Log.error('info');
````

repository: devFallingstar/MagicMirror

repo_name: devFallingstar/MagicMirror


text:
# Release Summary

This release focuses on the stabilization of Arktos as well as new features in multi-tenancy, scalability and unified VM/Container. Major improvements include:

* Multi-tenancy: virtualized multi-tenancy cluster based on short path and access control.
* Scalability: API server data partitioning and performance test in AWS.
* Unified VM/Container: Partial runtime services readiness and storage volume support.

# Key Features and Improvements

Multi-tenancy:

* Multi-tenancy design update [#101](https://github.com/futurewei-cloud/arktos/pull/101)
* Tenancy short-path support [#50](https://github.com/futurewei-cloud/arktos/pull/50)
* Add Tenant Controller [#124](https://github.com/futurewei-cloud/arktos/pull/124)
* Tenancy-aware token Authenticator [#129](https://github.com/futurewei-cloud/arktos/pull/129)
* Tenancy-aware Cert Authenticator [#99](https://github.com/futurewei-cloud/arktos/pull/99)
* Tenancy-aware RBAC Authorizer [#20](https://github.com/futurewei-cloud/arktos/pull/20)
* Tenancy in kubeconfig context [#69](https://github.com/futurewei-cloud/arktos/pull/69)
* Stabilization, more test and workaround fixes [#92](https://github.com/futurewei-cloud/arktos/pull/92)

Scalability:

* API Server Data Partitioning [#105](https://github.com/futurewei-cloud/arktos/pull/105),  [#65](https://github.com/futurewei-cloud/arktos/pull/65)
* Tools and guidance for setting up data partitioned Arktos cluster [#62](https://github.com/futurewei-cloud/arktos/pull/62)
* Add kube-up and start-kubemark for AWS [#127](https://github.com/futurewei-cloud/arktos/pull/127)

Unified VM/Container:

* Add support for primary runtime [#126](https://github.com/futurewei-cloud/arktos/pull/126)
* Add volume driver for OpenStack Cinder [#93](https://github.com/futurewei-cloud/arktos/pull/93)
* Fix issues on VM pod vCPU settings [#139](https://github.com/futurewei-cloud/arktos/pull/139)

repository: VonRosenchild/arktos

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: VonRosenchild/arktos


text:
Rule Set F

Finalized: May 17, 2015

* Kevin hosts every episode | HOST
* Each episode is roughly 1 hour long | LENGTH
* We record episodes using audio only | MEDIUM
* Episodes are posted every 3 weeks | FREQUENCY
* We choose an explicitly philosophical topic to guide every episode | TOPIC
* Digressions (aka. rabbit trails) are okay | FLAVOR
* We use post-show music at the end of each episode | MUSIC
* We include post-show banter cut from each episode | POSTSHOW
* We include a “stinger” at the end of each episode | STINGER
* We do not edit episodes for time and content | EDITING
* We assume listeners have heard at least 5 recent episodes | BACKGROUND
* We record using whatever devices are available | EQUIPMENT
* We record wherever is most convenient | LOCATION
* The host ends every episode with the phrase "We'll see you next time, on Bad Philosophy" | CLOSING
* For every episode, we must have at least one guest who is not like us (i.e. not white/male/straight/etc.) | GUEST
* If any of these rules are not met, each host may choose one song lyric that the other must post on their personal Twitter account within a day of the episode going live with the hashtag #bfail | FAIL
* For 5 minutes each episode, the hosts shall authentically embody someone they have met in person | IMITATE

license: MIT

repository: badphilosophy/Rules

repo_name: badphilosophy/Rules


text:
# LIRI Bot App

## Description
LIRI is like iPhone's SIRI. However, while SIRI is a Speech Interpretation and Recognition Interface, LIRI is a Language Interpretation and Recognition Interface. LIRI will be a command line node app that takes in parameters and gives you back data. LIRI will search Spotify for songs, Bands in Town for concerts, and OMDB for movies.

## Visuals

<!-- ![image] () -->

<!-- [![Watch the video](https://i.imgur.com/vKb2F1B.png)](https://youtu.be/vt5fpE0bzSY) -->

## Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

## Authors
### Meta
Paoge Moua - primary author
* Contact info<EMAIL_ADDRESS>OR<EMAIL_ADDRESS>* Portfolio address: https://paogemoua.github.io/
* GitHub: https://github.com/paogemoua

## License
Distributed under the [MIT] license.

## Reference

## Project Status
11.20.2018 Project open - in progress

<!-- Linked -->
[MIT]: https://choosealicense.com/licenses/mit/

repository: paogemoua/liri-node-app

repo_name: paogemoua/liri-node-app


text:
+++
Talk_date = ""
Talk_start_time = ""
Talk_end_time = ""
Title = "The CAP Theorem of Humans"
Type = "talk"
Speakers = ["christian-witts"]
slideshare = "https://www.slideshare.net/ChristianWitts/the-cap-theorem-of-humans"
speakerdeck = "https://speakerdeck.com/christianwitts/the-cap-theorem-of-humans"
youtube = "7fgOlAnZ0hk"
+++

No distributed system is safe from failures, and neither is your organisation. Is it possible to extend the CAP theorem to people and gain a Consistent, Available, and Partition Tolerant workforce in the face of disparate levels of leadership, anachronisms, and a million priorities ?

This talk will cover some of the ways you can enable your organisation to ensure that flying into the face of danger
shall not cause you fear.

repository: KrisBuytaert/devopsdays-web

gha_license_id: NOASSERTION

repo_name: KrisBuytaert/devopsdays-web


text:
# HandyFriends

## Table of Contents
1. [Overview](#Overview)
1. [Product Spec](#Product-Spec)
1. [Wireframes](#Wireframes)
2. [Schema](#Schema)

## Overview
### Description
An iOS app designed to make life easier for both customers and independent contractors. Customers can receive help real time from any independent contractor nearby by paying for the rates they charge. Receive service anywhere, at anytime.

### App Evaluation
- **Category:**
Contactless and Same Day Help
- **Mobile:**
iOS only
- **Story:**
An app for independent contractors to attract customers; Normal people can place an "order" for immediate help from anyone with the skills nearby. Contractors are able to charge their own rates and pick up jobs nearby real time.
- **Market:**
Contractors looking for more income. Contractors can be plumbers, electricians, mechanics and etc.
Customers are everyday citizens with maintenece issues at home.
- **Habit:**

- **Scope:**

## Product Spec

### 1. User Stories (Required and Optional)

**Required Must-have Stories**

* User logs in (or signs up) to access menu of various services they can get
* App remembers the user for certain period of time (automatic logout only after certain period of time)
* User creates a task: picks the service they need & give the required information (type of service, pictures, dates, max budget)
* User sees which contractors are available to help them & can choose one
* Once a task is created, user can cancel and edit the task
* Log of current and past tasks (split bar on top or a chronological list)
* Settings (Accessibility, Payment Info, Notification, General, etc.)

**Optional Nice-to-have Stories**

* Messenger: User can text a contractor.
* Login: verify email or phone number
* User can see when the contractor was online and whether they saw the message
* Settings: user can press the button to become a contractor
* Settings: referral code is generated (user can copy it or send it directly to someone)
* Settings: user can change password
*User can give a review & rate the contractor

### 2. Screen Archetypes

* [Initial screen]
   * When app is loading, user sees a static image with app logo

* [Login screen]
   * If user does not have an account, press sign up to create one
   * Support 3rd party authentication - etc ‘login with google’
   * Supports “remember me” feature; i.e. if user re-logins after certain period of time, he/she does not need to re-login

* [Register screen]
   * Upon Download/Reopening of the application, user is prompted to log in to gain access to their profile information to be properly matched with another     person.
   * User has an option to register with Google, etc.

* [Onboarding screen- Optional]
   * First-time user visual tutorial for how the app works

* [Task selection screen]
   * User can see a several scrollable icons representing various categories of tasks
   * Upon selecting one icon, user sees a list of more specific tasks (drop-down) and can choose one. Upon choosing, the user is “moved” to ‘Task Creation Screen’

* [Task creation screen]
   * User mush add task description (text)
   * User must upload relevant pictures (or take)
   * User must specify dates / times when the task can be completed
   * User must specify max budget (optional)
   * User must add location of the task (address)
   * User should estimate time the it will take one to finish the task (including start time if possible)
   * When all the necessary details are inputted, the User is directed to the ‘Choosing Contractor Screen’

* [Choosing Contractor Screen]
   * User is able to select contractor (from a list that can be updated by scrolling and potentially filtered) to complete the task
   * User can commit to using a specific contractor
   * (optional) User is able to chat with the contactor to ask clarifying questions using a button that direct user to “Messaging Screen”
   * (optional) fee is reserved (charged to CC at time of commitment)

* [Contractor rating screen - Optional]
   * can be accessed through ‘Current and Previous Tasks Screen’
   * User can leave review and assign rating to any contractor

* [Current and Previous Tasks Screen ]
   * User can see list of tasks in reverse chronological order
      *(Optional) User can filter tasks by certain features

   * User can click on any task; this action takes user to ‘View Task Details’ screen

### 3. Navigation

**Tab Navigation** (Tab to Screen)

* [Home]
Discover all the tasks and services available on the platform
* [Tasks]
View details of scheduled tasks and past tasks.
* [Taskers]
View past contractors you have worked with. Review which ones are your favorite
* [Profile]
Private information such as credit card, email and passwords can be found here

**Flow Navigation** (Screen to Screen)

* [list first screen here]
   * [list screen navigation here]
   * ...
* [list second screen here]
   * [list screen navigation here]
   * ...

## Wireframes
![Image from iOS](https://user-images.githubusercontent.com/61493372/99138178-a7ec0980-25e3-11eb-9991-256ccc0cd370.jpg)

### [BONUS] Digital Wireframes & Mockups

### [BONUS] Interactive Prototype

## Schema
[This section will be completed in Unit 9]
### Models
[Add table of models]
### Networking
- [Add list of network requests by screen ]
- [Create basic snippets for each Parse network request]
- [OPTIONAL: List endpoints if using existing API such as Yelp]

repository: gbarbachov/HandyFriends

branch_name: refs/heads/main

repo_name: gbarbachov/HandyFriends


text:
Team Dnntal

Ariel Cohen-Codar (ac4391)
Tomer Zwi (tz2247)
Suhyun Kim (ek2993)

1. Youtube Video: https://youtu.be/2eegzXgroKQ

2. Complete Code: https://github.com/gitskim/DnntalPrivate
3. Write-up: http://www.columbia.edu/cu/dnntal/dnntal_report.pdf

You can also access to the report from our flask application. To run it, pull the repository, do into the folder "Flaskapp".
Before running app.py, upload the model with the drive link we will send you in coursework. Put the folder models into the folder Flask.
Then run from the command shell : python app.py

You made need to install some dependencies. We could do a pip freeze but im the environment we worked with, there are more packages than those required so just pip install the dependencies you need. (One you may note have is "pip install imgaug".

Once you run the python app.py, the web app will run it the address indicated in the command shell.
This website present the projet and by clicking on learn more you will load the report.

If you want to test the app yourselve, we can send you a small sample of images but we don't have the authorization to share the dataset (yet..).

If you have any problem to run the app, send us an email<EMAIL_ADDRESS>
Also, you can visit the first landing page, without the model, we deployed on the following address (only you can't run the flask app and wait . for our reply):
http://www.columbia.edu/cu/dnntal/

We'd like to thank every teaching staff for the ADL class. We have learned so much and had a great time working on this project.

PS : Because of my plane to Paris, I had to leave whereas the video was yet to finish recording. That's why you don't hear my voice in the video (Ariel).

repository: gitskim/DnntalPrivate

repo_name: gitskim/DnntalPrivate


text:
# Contributing Guidelines

🎉 First of all, thanks for taking the time to contribute! 🎉

The following is a set of guidelines for contributing to this project. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.

 By participating, you are expected to uphold this code. Please report unacceptable behavior to [kaiwalyakoparkar@gmail.com](mailto:kaiwalyakoparkar@gmail.com)or [in our discord community](https://discord.com/invite/jvdcY2NkXa).

## How Can I Contribute?

### Reporting Bugs

This section guides you through submitting a bug report. Following these guidelines helps maintainers and the community understand your report 📝, reproduce the behavior 💻 💻, and find related reports 🔎.

Before creating bug reports, please check [this list](#before-submitting-a-bug-report) as you might find out that you don't need to create one. When you are creating a bug report, please [include as many details as possible](#how-do-i-submit-a-good-bug-report). Fill out [the required template](https://github.com/kaiwalyakoparkar/Lilliputian-Bot/blob/main/.github/ISSUE_TEMPLATE/bug_report.md), the information it asks for helps us resolve issues faster.

> **Note:** If you find a **Closed** issue that seems like it is the same thing that you're experiencing, open a new issue and include a link to the original issue in the body of your new one.

#### Before Submitting A Bug Report

- **Perform a [cursory search](https://github.com/kaiwalyakoparkar/Lilliputian-Bot/issues)** to see if the problem has already been reported. If it has **and the issue is still open**, add a comment to the existing issue instead of opening a new one.

#### How Do I Submit A (Good) Bug Report?

Bugs are tracked as [GitHub issues](https://guides.github.com/features/issues/). Create an issue on this repository and provide the following information by filling in [the template](https://github.com/kaiwalyakoparkar/Lilliputian-Bot/blob/main/.github/ISSUE_TEMPLATE/bug_report.md).

Explain the problem and include additional details to help maintainers reproduce the problem:

- **Use a clear and descriptive title** for the issue to identify the problem.
- **Describe the exact steps which reproduce the problem** in as many details as possible. **don't just say what you did, but explain how you did it**.
- **Provide specific examples to demonstrate the steps**. Include links to files or GitHub projects, or copy/pasteable snippets, which you use in those examples. If you're providing snippets in the issue, use [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines).
- **Describe the behavior you observed after following the steps** and point out what exactly is the problem with that behavior.
- **Explain which behavior you expected to see instead and why.**
- **Include screenshots and animated GIFs** which show you following the described steps and clearly demonstrate the problem. You can use [this tool](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [this tool](https://github.com/colinkeenan/silentcast) or [this tool](https://github.com/GNOME/byzanz) on Linux.

### Suggesting Enhancements

This section guides you through submitting an enhancement suggestion, including completely new features and minor improvements to existing functionality. Following these guidelines helps maintainers and the community understand your suggestion 📝 and find related suggestions 🔎.

Before creating enhancement suggestions, please check [this list](#before-submitting-an-enhancement-suggestion) as you might find out that you don't need to create one. When you are creating an enhancement suggestion, please [include as many details as possible](#how-do-i-submit-a-good-enhancement-suggestion). Fill in [the template](https://github.com/kaiwalyakoparkar/Lilliputian-Bot/blob/main/.github/ISSUE_TEMPLATE/feature-request.md), including the steps that you imagine you would take if the feature you're requesting existed.

#### Before Submitting An Enhancement Suggestion

- **Perform a [cursory search](https://github.com/kaiwalyakoparkar/Lilliputian-Bot/issues)** to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.

#### How Do I Submit A (Good) Enhancement Suggestion?

Enhancement suggestions are tracked as [GitHub issues](https://guides.github.com/features/issues/). Create an issue on that repository and provide the following information:

- **Use a clear and descriptive title** for the issue to identify the suggestion.
- **Provide a step-by-step description of the suggested enhancement** in as many details as possible.
- **Provide specific examples to demonstrate the steps**. Include copy/pasteable snippets which you use in those examples, as [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines).
- **Describe the current behavior** and **explain which behavior you expected to see instead** and why.
- **Include screenshots and animated GIFs** which help you demonstrate the steps or point out the part of this project which the suggestion is related to. You can use [this tool](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and [this tool](https://github.com/colinkeenan/silentcast) or [this tool](https://github.com/GNOME/byzanz) on Linux.
- **Explain why this enhancement would be useful** to most users and is something that can or should be implemented.

### Your First Code Contribution

Unsure where to begin contributing to Atom? You can start by looking through these `beginner` and `help-wanted` issues:

- [Beginner issues](https://github.com/kaiwalyakoparkar/Lilliputian-Bot/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) - issues which should only require a few lines of code, and a test or two.
- [Help wanted issues](https://github.com/kaiwalyakoparkar/Lilliputian-Bot/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) - issues which should be a bit more involved than `beginner` issues.

### Pull Requests

The process described here has several goals:

- Maintain the project's quality
- Fix problems that are important to users
- Engage the community in working toward the best possible!
- Enable a sustainable system for maintainers to review contributions

Please follow all instructions in [the template](https://github.com/kaiwalyakoparkar/Lilliputian-Bot/blob/main/.github/pull_request_template.md)

## Commit Guide
We follow the [conventional commits](https://www.conventionalcommits.org/) for our commit messages.

repository: kaiwalyakoparkar/one-api

gha_language: JavaScript

gha_license_id: MIT

repo_name: kaiwalyakoparkar/one-api


text:
# How to build

`go build` will let you build the source code.

Then you can `./server` to start the server without the assets and html loaded (you probably need to do this in production).

Or `./server -config=/PATH/TO/CONFIGURATION`to start the server with assets loaded (ideally in development environment).

repository: ReanimatingTheNew/koderunr

repo_name: ReanimatingTheNew/koderunr


text:
# Bttc.js on the Browser

To use Bttc.js on simple HTML projects, this directory contains some sample files

Open `index.html` in your browser and check console to get a bttc object. You can refer API docs to now use this object for function calls on the client side.

repository: panoptisDev/js-sdk

repo_name: panoptisDev/js-sdk


text:
# floop   [![Build Status](https://travis-ci.org/d3sw/floop.svg?branch=master)](https://travis-ci.org/d3sw/floop)

Floop is an integration tool which can be used to provide event-based lifecycle data for any process, without requiring any code changes which would couple your application to its consumer. It currently supports integration with STDOUT, HTTP and NATS endpoints, with near-term plans to add additional protocols.

Integration actions can be configured for the following lifecycle events:
* Begin
* Progress
* Completed
* Failed

## Examples

Floop is currently configured with a single configuration file.  The following example will post JSON to
a single endpoint when the `find` process starts and when it completes.

#### Running Floop

`floop reference_id=1234 -exec find . -name filename`

#### Configuring Floop

```
meta:
  - reference_id

handlers:
  begin:
  - type: http
    uri: "http://my-status-service/${Meta.REFERENCE_ID}"
    options:
      method: "POST"
    body: |
        {
            "status": "started",
        }
  completed:
  - type: http
    uri: "http://my-status-service/${Meta.REFERENCE_ID}"
    options:
      method: "POST"
    body: |
        {
            "status": "completed",
        }
```

Additional configuration examples can be found under the [test-data](/test-data) directory.

## Contributing

The purpose of this repository is to continue to evolve floop, adding additional integration protocols
and configuration options.  All development is done publicly on GitHub, and we look forward to working
with a community of both users and developers to make floop even more powerful and intuitive.

Please see the [Contributing Guide](CONTRIBUTING.md) for additional inforamtion on how to open issues
or submit pull requests.

gha_language: Go


text:
---
layout: post
title: How we sped up an ASP.NET Core endpoint from 20+ seconds down to 4 seconds
description: A deep dive into how ASP.NET Core MVC works around Newtonsoft.Json being synchronous.
---

_This post has initially been published on the Telstra Purple blog at <https://purple.telstra.com/blog/how-we-sped-up-an-aspnet-core-endpoint-from-20-seconds-down-to-4-seconds>._

# Introduction

We have an internal application at work that sends large payloads to the browser, approximately 25MB.

We know it is a problem and it is on our radar to do something about it.
In this article, we'll go through the investigation we performed, and how we ultimately brought the response time of this specific endpoint from 20+ seconds down to 4 seconds.

## The problem we faced

We run this application on an Azure App Service, and the offending endpoint had always been slow, and I personally assumed that it was due to the amount of data it was returning, until one day for testing purposes, I ran the app locally and noticed that it was much faster, between 6 and 7 seconds.

To make sure we were not comparing apples to oranges, we made sure that the conditions were as similar as they can be:

- We were running the same version of the app — that is, the same Git commit, we didn't go as far as running the exact same binaries;
- The apps were connecting to the same Azure SQL databases; and
- They were also using the same instance of Azure Cache for Redis.

The one big difference that we could see was that our dev laptops are much more powerful in regards to the CPU, the amount of RAM or the speed of the storage.

## The investigation

What could explain that this endpoint took roughly 3 times less to execute when it was connecting to the same resources?

To be perfectly honest, I can't remember exactly what pointed me in this direction, but at some point I realised two things:

1. Starting with ASP.NET Core 3.0, synchronous I/O is disabled by default, meaning an exception will be thrown, if you try to read the request body or write to the response body in a synchronous, blocking way; see the [official docs](https://docs.microsoft.com/en-us/dotnet/core/compatibility/aspnetcore#http-synchronous-io-disabled-in-all-servers) for more details on that; and
1. Newtonsoft.Json, also known as JSON.NET, is synchronous. The app used it as the new System.Text.Json stack didn't exist when it was migrated from ASP.NET Classic to ASP.NET Core.

How then did the framework manage to use a synchronous formatter while the default behaviour is to disable synchronous I/O, all without throwing exceptions?

I love reading code, it was then a great excuse for me to go and have a look at the implementation.
Following the function calls from [`AddNewtonsoftJson`](https://source.dot.net/#Microsoft.AspNetCore.Mvc.NewtonsoftJson/DependencyInjection/NewtonsoftJsonMvcBuilderExtensions.cs,abf21e3df206c817,references), we end up in the [`NewtonsoftJsonMvcOptionsSetup`](https://source.dot.net/#Microsoft.AspNetCore.Mvc.NewtonsoftJson/DependencyInjection/NewtonsoftJsonMvcOptionsSetup.cs,62) where we can see how we replace the System.Text.Json-based formatter for the one based on Newtonsoft.Json.

That specific formatter reveals it's performing some Stream gymnastics — see [the code on GitHub](https://github.com/dotnet/aspnetcore/blob/release/3.1/src/Mvc/Mvc.NewtonsoftJson/src/NewtonsoftJsonOutputFormatter.cs#L134-L165).
Instead of writing directly to the response body, the JSON.NET serializer writes (synchronously) to an intermediate `FileBufferingWriteStream` one, which is then used to write (asynchronously this time) to the response body.

The XML docs of [`FileBufferingWriteStream`](https://github.com/dotnet/aspnetcore/blob/a9449cd20c2150917355d8ba7a30fa19b47569f7/src/Http/WebUtilities/src/FileBufferingWriteStream.cs) explain it perfectly:

> A Stream that buffers content to be written to disk.
> Use `DrainBufferAsync(Stream, CancellationToken)` to write buffered content to a target Stream.

That Stream implementation will hold the data in memory while it's smaller than 32kB; any larger than that and it stores it in a temporary file.

If my investigation was correct, the response body would've been written [in blocks of 16kB](https://github.com/dotnet/aspnetcore/blob/a9449cd20c2150917355d8ba7a30fa19b47569f7/src/Mvc/Mvc.Core/src/Infrastructure/MemoryPoolHttpResponseStreamWriterFactory.cs#L28).
A quick math operation would show: 25MB written in 16kB blocks = 1,600 operations, 1,598 of which involve the file system. Eek!

This could explain why the endpoint was executing so much quicker on my dev laptop than on the live App Service; while my laptop has an SSD with near-immediate access times and super quick read/write operations, our current App Service Plan still runs with spinning disks!

How can we verify whether our hypothesis is correct?

## Solution #1, quick and dirty

The easiest way I could think of to get the file system out of the equation was to enable synchronous I/O.

```csharp
// Startup.cs
public void ConfigureServices(IServiceCollection services)
{
    services
        .AddControllers(options =>
        {
            // Suppress buffering through the file system
            options.SuppressOutputFormatterBuffering = true;
        })
        .AddNewtonsoftJson();
}

// Controller.cs
public Task<IActionResult> Action()
{
    // From https://docs.microsoft.com/en-us/dotnet/core/compatibility/aspnetcore#http-synchronous-io-disabled-in-all-servers
    // Allow synchronous I/O on a per-endpoint basis
    var syncIOFeature = HttpContext.Features.Get<IHttpBodyControlFeature>();
    if (syncIOFeature != null)
    {
        syncIOFeature.AllowSynchronousIO = true;
    }

    // Rest of the implementation, ommited for brevity
}
```

Making both of those changes is required because:

1. Only suppressing output buffering would throw an exception, since we'd be synchronously writing to the response body, while it's disabled by default;
1. Only allow synchronous I/O wouldn't change anything, as output buffering is enabled by default, so that updating projects to ASP.NET Core 3.0 doesn't break when using Newtonsoft.Json and sending responses larger than 32kB.

Locally, I observed a response time of ~4 seconds, which was a substantial improvement of ~30%.

While it was a good sign that our hypothesis was correct, we didn't want to ship this version.
Our application doesn't get that much traffic, but synchronous I/O should be avoided if possible, as it is a blocking operation that can lead to thread starvation.

## Solution #2, more involved, and more sustainable

The second option was to remove the dependency on Newtonsoft.Json, and use the new System.Text.Json serialiser.
The latter is async friendly, meaning it can write directly to the response stream, without an intermediary buffer.

It wasn't as easy as swapping serialisers, as at the time of writing System.Text.Json it was not at feature parity with Newtonsoft.Json.
My opinion is that it's totally understandable as JSON.NET has been available far longer.

Microsoft provides a good and honest comparison between the two frameworks: <https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-migrate-from-newtonsoft-how-to>

The main thing for us was that System.Text.Json doesn't support ignoring properties with default values. For example: like `0` for integers.
We couldn't just ignore this, because the payload is already so large, adding unnecessary properties to it would have made it even larger.
Luckily, the workaround was relatively straightforward and [well documented](https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-migrate-from-newtonsoft-how-to#conditionally-ignore-a-property): we needed to write a custom converter, which gave us total control over which properties were serialised.

It was boring, plumbing code to write, but it was easy.

In the end, we removed the changes mentioned above, and plugged in System.Text.Json.

```csharp
// Startup.cs
public void ConfigureServices(IServiceCollection services)
{
    services
        .AddControllers()
        .AddJsonOptions(options =>
        {
            // Converters for types we own
            options.JsonSerializerOptions.Converters.Add(new FirstCustomConverter());
            options.JsonSerializerOptions.Converters.Add(new SecondCustomConverter());

            // Make sure that enums are serialised as strings by default
            options.JsonSerializerOptions.Converters.Add(new JsonStringEnumConverter());

            // Ignore `null` values by default to reduce the payload size
            options.JsonSerializerOptions.IgnoreNullValues = true;
        });
}
```

We gave it another go, and again achieved a consistent 4-second response time on that endpoint 🎉.
After deploying this new version to our App Service, we were stoked to see a similar response time there as well.

![App Insights timings](/public/images/posts/2020-06-06-how-we-sped-up-an-endpoint-from-20-seconds-down-to-4/app-insights-timings.png)

## Conclusion

This was a super fun investigation, and I was once again super happy to dig into the ASP.NET Core internals and learn a bit more about how some of it works.

While I realise our case was extreme given the size of the response payload, in the future I'll think twice when I encounter a code base using Newtonsoft.Json, and see how hard it'd be to move to System.Text.Json.
There's definitely a big gap between both, but the team is hard at work to fill some of it for .NET 5.

See the [.NET 5 preview 4 announcement](https://devblogs.microsoft.com/dotnet/announcing-net-5-preview-4-and-our-journey-to-one-net/) and search for the "Improving migration from Newtonsoft.Json to System.Text.Json" header to learn more about where the effort goes.
You can also check [issues with the "area-System.Text.Json"](https://github.com/dotnet/runtime/labels/area-System.Text.Json) on the `dotnet/runtime` repository, or take a look at [the specific project board for System.Text.Json](https://github.com/dotnet/runtime/projects/25) if you want to be even closer from what's happening.

gha_language: CSS


text:
---
layout: post
title: Developers, Love Your SQL Database
date: 2018-03-12 19:30:15.000000000 +00:00
type: post
parent_id: '0'
published: true
password: ''
status: publish
categories: []
tags: []
meta:
  _wpcom_is_markdown: '1'
  _rest_api_published: '1'
  _rest_api_client_id: "-1"
  _publicize_job_id: '15656264083'
  timeline_notification: '1520883016'
author:
  login: timabell
  email<EMAIL_ADDRESS>  display_name: timabell
  first_name: ''
  last_name: ''
permalink: "/2018/03/12/developers-love-your-sql-database/"
---
Developers, don't be afraid of your SQL database, don't try and ignore it, try and make it the best it can be just like you do for your code.

## What's wrong today

I've noticed that a lot of projects that have SQL databases don't apply the high standards that they have for their codebase (refactoring, clean code, documentation, etc.) to the SQL part of their projects.

*   Having to handle live data makes refactoring harder (but not impossible).
*   SQL is a language from the stone-age of software.

But that's no excuse for not doing the best you can for the problem at hand.

## There are answers to these problems

Some of the answer is just accepting that it's harder and learning to be effective. For example I find [VsVim](https://marketplace.visualstudio.com/items?itemName=JaredParMSFT.VsVim) and [SQL Prompt](https://www.red-gate.com/products/sql-development/sql-prompt/) to be great for editing SQL.

[Redgate](https://www.red-gate.com/) have a great suite of tools for managing the full lifecycle of a database. Use them. I particularly like the way [ReadyRoll](https://www.red-gate.com/products/sql-development/readyroll/) handles migrations. RedGate tooling has excellent [integrations with Octopus deploy](https://octopus.com/blog/database-deployments-with-octopus-and-redgate-sql-release).

So if you're in the situation I've seen with a database you virtually can't change and nothing but a visual studio database project in place that may or may not be in place with production, then try the following:

1.  Throw away the database project.
2.  Set up ReadyRoll in your visual studio project.
3.  Grab your production schema, and use that as a base in your ReadyRoll migrations. (It has a neat feature for a first "migration" that will only be run on a new database build, so it can build you a dev database from scratch, but it won't attempt to recreate production).
4.  Configure ready-roll to use branch folders.
5.  Start creating migrations in feature branches.
6.  Set up CI / test / QA / pre-prod environments (or whatever you call them in your team) to drop and rebuild the database with every build. You can make this faster by using SQL Server snapshots.
7.  Each environment then runs the database migrations that match the version of the codebase that was released to that environment.
8.  Push changes up through the environments until those migration scripts have been run so many times you are no longer afraid to run them.

There are many potential complications, such as re-indexing tables with lots of data in production, but these are not things that an intelligent well-functioning DevOps team can't handle well once the basic process is in place.

## Further Reading

*   [https://gist.github.com/timabell/6fbd85431925b5724d2f](https://gist.github.com/timabell/6fbd85431925b5724d2f) - source control your schema documentation (ms_description attributes)
*   [https://dba.stackexchange.com/questions/515/how-do-you-document-your-databases](https://dba.stackexchange.com/q/515/33693)

* * *

If you like this, you might appreciate the tool I'm working on to shed further light on the dark corners of your SQL database, check it out at [http://schemaexplorer.io/](http://schemaexplorer.io/?utm_source=blog.timwise&utm_medium=web&utm_campaign=love-db) and be sure to sign up to the waiting list.

gha_language: SCSS


text:
# Kafka Axon Springboot Example

This is an example SpringBoot application using the Kafka Axon extension.
It configures a simple Kafka message publishing using Kafka infrastructure run locally.

## How to run

### Preparation

You will need `docker` and `docker-compose` to run this example.

Please run:

```bash
docker-compose -f ./kafka-axon-example/docker-compose.yaml up -d
```

This will start [Zookeeper](https://zookeeper.apache.org/), [Kafka](https://github.com/wurstmeister/kafka-docker),
 [KafkaCat](https://github.com/edenhill/kafkacat), [Kafka Rest](https://github.com/nodefluent/kafka-rest) and [Kafka Rest UI](https://github.com/nodefluent/kafka-rest-ui).
KafkaCat can be used to investigate the setup, whereas the UI (accessed through localhost:8000, user `admin` and password `admin`) provides visualization of the internals.

Now build the application by running:

```bash
mvn clean package -f ./kafka-axon-example
```

### Running example application

You can start the application by running `java -jar ./kafka-axon-example/target/axon-kafka-example.jar`.

From a Kafka Message Source perspective, there are several options you have, as both consumption and production of
event messages can be Subscribing or Streaming (aka push or pull).
Thus, the application can run in six different modes due to the possibility to define a producer
 and consumer event processing mode.
At this stage the following profiles can be used:

  * `subscribing-producer`
  * `tracking-producer`
  * `pooled-streaming-producer`
  * `subscribing-consumer`
  * `tracking-consumer`
  * `pooled-streaming-consumer`

If not specified, a `subscribing` producer and `tracking` consumer will be used.
To activate these modes, please use Spring profiles in the run configuration like so:
 `--spring.profiles.active=tracking-producer,subscribing-consumer`

repository: help-lixin/extension-kafka

repo_name: help-lixin/extension-kafka


text:
## Flush console input

---

### About `flushconin`

- Use `flushconin` to flush console input buffer.
- Works only for Windows.
- [Documentation: godoc](https://godoc.org/github.com/jeet-parekh/flushconin)

---

- `FlushConsoleInput()` returns an error if unsuccessful.
- `MustFlushConsoleInput()` panics if unsuccessful.

---

repository: jeet-parekh/flushconin

repo_name: jeet-parekh/flushconin


text:
---
layout: post
title: "Opcode Script"
description: "Opcodes from Assembly Instructions"
tags: [asm, opcodes,]
---

While working through some of the [corelan.be](https://www.corelan.be)
tutorials I came across the "assemble" functionality in windbg which
allows you to lookup the opcodes of assembly instructions. I have
previously used an online assembler for this sort of functionality
before located at
[defuse.ca](https://defuse.ca/online-x86-assembler.htm) but I wanted
the ability to lookup the instructions on the command line without
going to the web. I looked into the functionality of `gdb` and `nasm`
but it doesn't appear that either provide this sort of functionality
so...

Here is a little shell function I put in my `.bashrc`:

{% highlight bash %}
function asmop {
DIR=/tmp/asmop
mkdir $DIR
cd $DIR
FILE=tmpasm
while read x ; do
  touch $FILE.nasm
cat << EOF > $FILE.nasm
global _start
section .text
_start:
EOF
  echo $x >> $FILE.nasm;
  nasm -f elf32 -o $FILE.o $FILE.nasm
  ld -o $FILE $FILE.o
  objdump -D $FILE -M intel
  rm $FILE*
done
cd ../
rmdir $DIR
}
{% endhighlight %}

This lets us lookup opcodes for assembly like:

{% highlight bash %}
~/$ asmop
jmp esp

tmpasm:     file format elf32-i386

Disassembly of section .text:

08048060 <_start>:
 8048060:	ff e4                	jmp    esp
push eax

tmpasm:     file format elf32-i386

Disassembly of section .text:

08048060 <_start>:
 8048060:	50                   	push   eax
{% endhighlight %}

repository: blischalk/blischalk-jekyll

repo_name: blischalk/blischalk-jekyll


text:
# go-containerregistry

[![Build Status](https://travis-ci.org/google/go-containerregistry.svg?branch=master)](https://travis-ci.org/google/go-containerregistry)
[![GoDoc](https://godoc.org/github.com/google/go-containerregistry?status.svg)](https://godoc.org/github.com/google/go-containerregistry)
[![Go Report Card](https://goreportcard.com/badge/google/go-containerregistry)](https://goreportcard.com/report/google/go-containerregistry)
[![Code Coverage](https://codecov.io/gh/google/go-containerregistry/branch/master/graph/badge.svg)](https://codecov.io/gh/google/go-containerregistry)

## Introduction

This is a golang library for working with container registries.
It's largely based on the [Python library of the same name](https://github.com/google/containerregistry).

The following diagram shows the main types that this library handles.
![OCI image representation](images/ociimage.jpeg)

## Tools

This repo hosts some tools built on top of the library.

### `crane`

[`crane`](cmd/crane/doc/crane.md) is a tool for interacting with remote images
and registries.

### `gcrane`

[`gcrane`](cmd/gcrane/README.md) is a GCR-specific variant of `crane` that has
richer output for the `ls` subcommand and some basic garbage collection support.

### `k8schain`

[`k8schain`](pkg/authn/k8schain/README.md) implements the authentication
semantics use by kubelets in a way that is easily consumable by this library.

`k8schain` is not a standalone tool, but it's linked here for visibility.

### Emeritus: [`ko`](https://github.com/google/ko)

This tool was originally developed in this repo but has since been moved to its
own repo.

repository: xiaohegong/go-containerregistry

repo_name: xiaohegong/go-containerregistry


text:
# The Variational State Interface

A central element of NetKet is the {class}`~netket.vqs.VariationalState` interface.
A Variational State represents an approximate, variational description of a system, and can be used to
probe it.

As of now, NetKet has two types of Variational state implementations:

- {class}`~netket.vqs.MCState`, which is a classical variational approximation of a pure state.
- {class}`~netket.vqs.MCMixedState`, which is a classical variational approximation of a mixed state.
- {class}`~netket.vqs.FullSumState`, which is a version of {class}`~netket.vqs.MCState` without sampling, performing the full summation over the whole Hilbert space and computing expectation values and gradients exactly.

It is our plan to build upon this interface for new kind of variational states in the future. For example, we
are interested in implementing a new kind of variational state that encodes a state into a Qiskit circuit.

## Constructing a Variational State

To construct the two variational states above you need to provide at least a Monte Carlo sampler (you can find the list of available ones [here](netket_sampler_api)) and a model.
The Hilbert space of the variational state will be inferred from the sampler, which contains a reference to the underlying Hilbert space.

The model can be specified in several ways. The most standard way is to pass a [Flax Linen Module](https://flax.readthedocs.io/en/latest/flax.linen.html#module), but you can also pass a Jax-style pair
of functions {code}`(init, apply)` or an haiku module obtained by calling {code}`haiku.transform()`.
It's also easy to use another jax-based framework with NetKet (though it's not yet documented). If you
want to do that, you can have a look at the automatic conversion [code](https://github.com/netket/netket/tree/master/netket/utils/model_frameworks) and get in touch with us.

:::{warning}
Initializing a Variational State with non-Flax modules is still experimental. Complex modules,
especially if mutable, might not work. If you have problems don't hesitate opening issues on
GitHub so that we can address them. They are mostly easy fixes.
:::

### Defining models: Flax

In general, the most-well supported way to define Models for NetKet is through Flax.

If you are not familiar with Flax, it's a package that allows to define Neural Networks or other
parametrized functions. While we suggest to read carefully the [introduction to Flax](https://flax.readthedocs.io/en/latest/notebooks/flax_basics.html), if you are impatient and just want to define some simple, easy
models you can find some examples in the [Model Surgery](https://flax.readthedocs.io/en/latest/howtos/model_surgery.html) section of Flax documentation.

Other examples can be found in the source of the [Pre-built models](netket_models_api) distributed with NetKet, such as
{class}`~netket.models.RBM` (the simplest one), {class}`~netket.models.MPSPeriodic` and {class}`~netket.models.NDM` for
more complicated examples.

:::{warning}
Flax modules do not always work well with complex numbers. For that reason, if your model has complex
parameters we invite you to use {code}`netket.nn` in place of {code}`flax.linen` and {code}`flax.nn`.

{code}`netket.nn` re-exports some Flax functionality ensuring that it works nicely with complex numbers.
If you don't find something from Flax in {code}`netket.nn`, please [open an issue](https://github.com/netket/netket/issues) in GitHub.
:::

### MCState

Let's now assume we have a Flax module, {class}`~netket.models.RBM` and we want to construct a variational state
on sampled with a simple {func}`~netket.sampler.MetropolisLocal` sampler.

To do this, first you need to construct the model and the sampler, and then you can pass them, together with several
parameters, to the constructor.

```python
hilbert = nk.hilbert.Spin(0.5)**10

sampler = nk.sampler.MetropolisLocal(hilbert)

model = nk.models.RBM(alpha=1, param_dtype=float, kernel_init=nn.initializers.normal(stddev=0.01))

vstate = nk.vqs.MCState(sampler, model, n_samples=500)
```

When constructed, the variational state will call the model's init method to generate the state and the
parameters, and will also initialize the sampler.

When constructing a variational state you can also pass a seed, that will be used to initialize both the
weights and the sampler. You can also pass two different seeds for the sampler and the weights.

(warn-chunking)=

:::{warning}
Sometimes you might be working on very large models that run on GPUs, or you might want to compute expectation values with a very high number of samples, which might cause out-of-memory errors. To prevent this, {code}`MCState` exposes the attribute {py:attr}`~netket.vqs.MCState.chunk_size`, which controls the size of the chunks used in the forward and backward computation of expectation values.

When this value is set, computations are not performed on huge matrices, but instead we split the inputs into smaller matrices and loop through them. In general, setting {py:attr}`~netket.vqs.MCState.chunk_size` to a low value will address the majority of memory issues, but will increase the computational cost.

A value of at least 128 is suggested, but will greatly depend on your model. You should try to use the largest value you can for your system. Setting it to {code}`None` is equivalent to setting it to infinity.
:::

## Using a Monte Carlo Variational State

### Expectation values

One you have a variational state, you can do many things with it.
First of all, you can probe expectation values:

```
Ĥ = nk.operator.Ising(hilbert, nk.graph.Chain(hilbert.size), h=0.5)

vstate.expect(Ĥ)

>>> -4.98 ± 0.14 [σ²=9.51, R̂=1.0006]
```

Notice that if you call multiple times {code}`expect`, the same set of
samples will be used, and you will get the same result. To force sampling
to happen again, you can call {py:meth}`~netket.vqs.MCState.sample`.

```
vstate.expect(Ĥ)

>>> -4.98 ± 0.14 [σ²=9.51, R̂=1.0006]

vstate.sample();

vstate.expect(Ĥ)

>>> -4.90 ± 0.14 [σ²=9.54, R̂=1.0062]
```

The set of the last sampled samples can be accessed from the attribute
{py:attr}`~netket.vqs.MCState.samples`. If you access the samples
from this attribute, but you haven't sampled already, {code}`sample()` will
be called automatically.

Note that when you update the parameters, the samples are automatically
discarded.

Parameters can be accessed through the attribute {py:attr}`~netket.vqs.VariationalState.parameters`,
and you can modify them by assigning a new set of parameters to this attribute.

Note that parameters cannot in general be modified in place, as they are
of type `FrozenDict` (they are frozen, aka can't be modified). A typical way
to modify them, for example to add 0.1 to all parameters is to do the following:

```python
import jax

# See the value of some parameters
vstate.parameters['visible_bias']

>>> DeviceArray([-0.10806808, -0.14987472,  0.13069461,  0.0125838 ,
              0.06278384,  0.00275547,  0.05843748,  0.07516951,
              0.21897993, -0.01632223], dtype=float64)

vstate.parameters = jax.tree_map(lambda x: x+0.1, vstate.parameters)

# Look at the new values
vstate.parameters['visible_bias']
>>> DeviceArray([-0.00806808, -0.04987472,  0.23069461,  0.1125838 ,
              0.16278384,  0.10275547,  0.15843748,  0.17516951,
              0.31897993,  0.08367777], dtype=float64)
```

### Sampling

You can also change the number of samples to extract (note: this will
trigger recompilation of the sample function, so you should not do this
in a hot loop) by changing {py:attr}`~netket.vqs.MCState.n_samples`, and
the number of discarded samples at the beginning of every markov chain by
changing {py:attr}`~netket.vqs.MCState.n_discard_per_chain`.

By default, {py:attr}`~netket.vqs.MCState.n_discard_per_chain` is 10% of
{py:attr}`~netket.vqs.MCState.n_samples`.

The number of samples is then split among the number of chains/batches of the sampler.

```python
hilbert = nk.hilbert.Spin(0.5)**6

sampler = nk.sampler.MetropolisLocal(hilbert, n_chains=8)

vstate = nk.vqs.MCState(sampler, nk.models.RBM(), n_samples=500)

print(vstate.n_samples)
504

print(vstate.chain_length)
63

print(vstate.n_discard_per_chain)
50
```

You can see that 500 samples are split among 8 chains, giving $500/8=62.5$ (rounded to
the next largest integer, 63). Therefore 8 chains of length 63 will be run.
n_discard_per_chain gives the number of discarded steps taken in the markov chain before actually storing
them, so the Markov Chains are actually {code}`chain_length + n_discard_per_chain` long. The default
n_discard_per_chain is 10% of the total samples, but you can change that to any number.

(warn-mpi-sampling)=

:::{warning}
When running your code under MPI, the length of the chain is computed not only by dividing the
total number of samples by {code}`n_chains`, but also by diving it by the number of MPI processes.

Therefore, considering the number from the example above, if we had 4 MPI processes, we would have
found a chain length of $500/(8*4) = 15.625 \rightarrow 16$.
:::

### Collecting the state-vector

A variational state can be evaluated on the whole Hilbert space in order to obtain
the ket it represents.

This is achieved by using the {py:meth}`~netket.vqs.VariationalState.to_array` method,
which by defaults normalises the $L_2$ norm of the vector to 1 (but can be turned off).

Mixed state Ansätze can be converted to their matrix representation with
{py:meth}`~netket.vqs.MCMixedState.to_matrix`. In this case, the default
normalisation sets the trace to 1.

### Manipulating the parameters

You can access the parameters of a variational state through the {py:attr}`~netket.vqs.VariationalState.parameters` attribute.
Similarly, if your model has also a mutable state, you can access it through
the {py:attr}`~netket.vqs.VariationalState.model_state` attribute.

Parameters are stored as a Flax {code}`FrozenDict`, which behaves like a standard python dictionary but cannot be modified.
In Jax jargon, Parameters are a PyTree (see [PyTree documentation](https://jax.readthedocs.io/en/latest/pytrees.html)) and they
can be operated upon with functions like [jax.tree_map](https://jax.readthedocs.io/en/latest/jax.tree_util.html?highlight=tree_map#jax.tree_util.tree_map).

You can also modify the parameters by \_unfreezing\_ them, using the command `flax.core.unfreeze`.

```python
import flax

pars = flax.core.unfreeze(varstate.parameters)

pars['Dense']['kernel'] = pars['Dense']['kernel'] +1

varstate.parameters = pars
```

The parameter dict will be automatically frozen upon assignment.

## Saving and Loading a Variational State

Variational States conform to the [Flax serialization interface](https://flax.readthedocs.io/en/latest/flax.serialization.html) and can be serialized and deserialized with it.

Moreover, the Json Logger {class}`~netket.logging.JsonLog` serializes the variational state
through that interface.

A simple example to serialize data is provided below:

```python
# construct an RBM model on 10 spins
vstate = nk.vqs.MCState(nk.sampler.MetropolisLocal(nk.hilbert.Spin(0.5)**10),
                                nk.models.RBM())

import flax

with open("test.mpack", 'wb') as file:
  file.write(flax.serialization.to_bytes(vstate))
```

And here we de-serialize it:

```python
# construct a new RBM model on 10 spins
vstate = nk.vqs.MCState(nk.sampler.MetropolisLocal(nk.hilbert.Spin(0.5)**10),
                                nk.models.RBM())

# load
with open("test.mpack", 'rb') as file:
  vstate = flax.serialization.from_bytes(vstate, file.read())
```

Note that this also serializes the state of the sampler.

:::{note}
The JSonLog serializer only serializes the parameters of the model, and not the whole variational state.
Therefore, if you wish to reload the parameters of a variational state, saved by the json logger, you should
use the same procedure outlined above, only that the list line should be:

```python
with open("parameters.mpack", 'rb') as file:
  vstate.variables = flax.serialization.from_bytes(vstate.variables, file.read())
```
:::

## Using FullSumState for testing

In order to check the performance of a given model on small test systems, it is possible to use {class}`~netket.vqs.FullSumState` which can be used in place of {class}`~netket.vqs.MCState` and computes quantities not by stochastic sampling but by evaluation of the ansatz over the full Hilbert space basis. (Therefore, it is infeasible to use for system sizes beyond ED.)
This can be helpful to diagnose potential errors arising from Monte Carlo sampling.

```python
> vs = nk.vqs.FullSumState(hilbert, nk.models.RBM())
```

repository: netket/netket

gha_language: Python

repo_name: netket/netket


text:
---
title: "Breaking out of your comfort zone"
date: "2017-04-16"
---

![Comfort Zone](./comfort-zone.jpg)

### Introduction

Welcome back guys, the last few posts has focused on more the technical side but this week will be a bit more personal. I will be discussing the Comfort Zone. So what is the "comfort zone"? I am pretty sure everyone is familiar with the concept of a comfort zone but this image sums it well:

> A **comfort zone** is a [psychological state](https://en.wikipedia.org/wiki/Mental_status_examination "Mental status examination") in which things feel familiar to a person and they are at ease and in control of their environment, experiencing low levels of [anxiety](https://en.wikipedia.org/wiki/Anxiety "Anxiety") and [stress](<https://en.wikipedia.org/wiki/Stress_(psychological)> "Stress (psychological)"). In this zone, a steady level of performance is possible - Wikipedia

While most people know the concept of a comfort zone, sometimes it is very difficult to identify what is inside and outside your comfort zone. There are obvious signs for certain comfort zones but for others, it's very hard. For example, it might be very easy to know places where you feel comfortable or people you are comfortable with but it's harder to identify your comfort zone in other areas such as your learning. This is the area I want to focus on in this post.

### My learning comfort zone

One area where I find myself constantly battling my comfort zone is with programming languages. Once you get used to programming in a certain way with a particular programming language it's very hard to break out of it. An example of this for me would be my relationship with the language Python.

At the start, I didn't take any courses on Python, I just pick up the language as I went along. I slowly created more and more complicated projects with python but I was always programming using the same techniques. I never heard of the concept of PEP, generators or decorators, I was:

> Programming **with** the language and not **in** the language - _Random person online_

In other words, I wasn't using the tools and features the language provided instead I was using it as if it was another programming language. \[I plan to write a bigger post on this subject\]

This finally changed when I was at an interview and I was asked to write a generator and I hadn't got a clue. In fairness to the interviewer, he explained exactly what it was and I got it. After that interview, I went on a Python tutorial binge and realised I was only using <20% of the language.

This was the situation that caused me to look at my learning and realised just how comfortable I was with certain tools and frameworks.

### Why going outside your comfort zone is a good thing?

So we discussed the comfort zone and I gave you an example of my own realisation of my comfort zone, why do we need to get out of our comfort zones?

- You learn new things
- You realise your weakness and how to improve them
- By breaking your comfort zone you might encourage others to do the same.
- The more you do it, the easier it gets and the more you learn about yourself

### Tips and Tricks for getting out of your comfort zone

We know the benefits and why we should go outside our comfort zone but how do we do it? Here are some tips I found very useful for breaking out of your comfort zone both in personal terms and programming:

- As soon as your brain is telling you reasons to not do something or you start feeling nervous don't think about it, just do it.   This might seem pointless but you realise how much of our lives are spent in our comfort zone.
- Read the docs for programming languages and look for more advanced tutorials on your chosen programming language.
- If the programming language you are using has a standard, read and it and code to it. The only exception to this is if your team has a very specific way they code then stick to that.

### Conclusion

Comfort zones are weird things...if you are aware of your comfort zones and how to break them, when you break them they can be life-changing. Anyway, I hope you enjoyed this post.

Until next time,

Jason!

repository: jaslloyd/my-blog

repo_name: jaslloyd/my-blog


text:
# Chaining

In this chapter we look at JavaScript’s ability to chain methods together. By using a few simple techniques, application developers can streamline their code authoring. As well as writing time-saving functions that reduce the burden of common tasks, you can improve how code is implemented. In the end, you can write an entire JavaScript library that embraces the tech- nique of chaining, and chains together all your favorite methods.
Chaining is really just a syntax hack. It allows you to express complex operations in a small amount of code by reusing an initial operation. Chaining requires two parts: a factory that creates an object around an HTML element (we cover factories in depth in Chapter 7), and methods that perform some action using that HTML element. Each of these methods can be added to the chain by appending the method name with a dot. Chaining can be thought of as the process of selecting an element on the page and performing one or more operations on it.
Let’s take a look at a small example. Using some predefined utility functions, you can get a basic idea of the concept of chaining by seeing a “before and after” contrast. How these utility functions work isn’t important for this illustration. This first example gets a reference to an element with the ID of example and assigns an event listener to it. When clicked, it sets the text color style to green, then shows the element:

```javascript
// Without chaining:
addEvent($('example'), 'click', function() {
  setStyle(this, 'color', 'green');
  show(this);
});
// With chaining:
$('example').addEvent('click', function() {
  $(this)
    .setStyle('color', 'green')
    .show();
});
```

## The Structure of a Chain

You are already familiar with the dollar function, which usually returns an HTML element or a collection of HTML elements as shown here:

```javascript
function $() {
  var elements = [];
  for (var i = 0, len = arguments.length; i < len; ++i) {
    var element = arguments[i];
    if (typeof element === 'string') {
      element = document.getElementById(element);
    }
    if (arguments.length === 1) {
      return element;
    }
    elements.push(element);
  }
  return elements;
}
```

However, if you modify the function to act as a constructor, store the elements as an array in an instance property, then return a reference to the instance in all prototype methods, you can give it the ability to chain. But let’s not get ahead of ourselves. You need to modify the dollar function so it becomes a factory method, creating an object that will support chaining. You also want the dollar function to be able to take in an array of elements, so you can use the same public interface as before. The modified code would look like this:

```javascript
(function() {
  // Use a private class.
  function _$(els) {
    this.elements = [];
    for (var i = 0, len = els.length; i < len; ++i) {
      var element = els[i];
      if (typeof element === 'string') {
        element = document.getElementById(element);
      }
      this.elements.push(element);
    }
  }
  // The public interface remains the same.
  window.$ = function() {
    return new _$(arguments);
  };
})();
```

Since all objects inherit from their prototype, you can take advantage of the reference to the instance object being returned and run each of the methods attached to the prototype as a chain. With that in mind, let’s go ahead and add methods to the private dollar constructor prototype. This will make chaining possible:

```javascript
(function() {
  function _$(els) {
    ...
  }
  _$.prototype = {
    each: function(fn) {
      for ( var i = 0, len = this.elements.length; i < len; ++i ) {
        fn.call(this, this.elements[i]);
      }
      return this;
    },
    setStyle: function(prop, val) {
      this.each(function(el) {
        el.style[prop] = val;
      });
      return this;
    },
    show: function() {
      var that = this;
      this.each(function(el) {
        that.setStyle('display', 'block');
      });
      return this;
    },
    addEvent: function(type, fn) {
      var add = function(el) {
        if (window.addEventListener) {
          el.addEventListener(type, fn, false);
        }
        else if (window.attachEvent) {
          el.attachEvent('on'+type, fn);
        }
      };
      this.each(function(el) {
add(el); });
      return this;
    }
  };
  window.$ = function() {
    return new _$(arguments);
  };
})();
```

If you examine the last line in each method of the class, you’ll notice that they all end in return this. This passes on the object to the next method in the chain. With a chainable interface, the possibilities are limitless. You can now start writing code like this:

```javascript
$(window).addEvent('load', function() {
  $('test-1', 'test-2')
    .show()
    .setStyle('color', 'red')
    .addEvent('click', function(e) {
      $(this).setStyle('color', 'green');
    });
});
```

This will attach an event to the window object’s load event. Upon firing, the elements with the IDs of test-1 and test-2 will instantly be shown, and the text within them will be set to the color red. They will then have click event listeners attached to them, which on firing will set the text color to green. That’s quite a bit packed into such a small amount of application code.
For those familiar with the jQuery JavaScript library, this interface is very similar. The anchor of the chain is the window object or an HTML element, and every operation is attached to that anchor. In the previous example, there are two chains: one that attaches the load event to the window object, and one that sets styles and attaches events to the elements with the IDs test-1 and test-2. Almost any set of existing utilities can be adapted to chaining using this style. We cover this more in the next section.

## Building a Chainable JavaScript Library

So far you’ve chained just a few of the most commonly used utility functions, but you can expand this to your heart’s desire. Building a JavaScript library takes much care and thought. It need not be thousands or even hundreds of lines of code; the length depends on what you need out of a library. You can look at some of the most common features that JavaScript libraries offer, and take it from there. The fundamentals that you will find in nearly all JavaScript libraries are shown in Table 6-1.

**Table 6-1** The Common Features Found in Most JavaScript Libraries
| Feature | Description                                                 |
| ------- | :---------------------------------------------------------- |
| Events  | Adding and removing listeners; normalizing the event object |
| DOM     | Class name management; style management                     |
| Ajax    | Normalizing XMLHttpRequest                                  |

If you were to build this out into pseudocode on top of the private dollar constructor, it might look something like this:

```javascript
// Include syntactic sugar to help the development of our interface.
Function.prototype.method = function(name, fn) {
  this.prototype[name] = fn;
  return this;
};

(function() {
  function _$(els) {
    ...
  }
  /* Events
   * addEvent
   * getEvent */
  _$.method('addEvent', function(type, fn) {
    ...
  }).method('getEvent', function(e) {
    ...
  }). /*
    DOM
      * addClass
      * removeClass
      * replaceClass
      * hasClass
      * getStyle
      * setStyle
  */
  method('addClass', function(className) {
    ...
  }).method('removeClass', function(className) {
    ...
  }).method('replaceClass', function(oldClass, newClass) {
    ...
  }).method('hasClass', function(className) {
    ...
  }).method('getStyle', function(prop) {
    ...
  }).method('setStyle', function(prop, val) {
    ...
  }).
  /* AJAX load. Fetches an HTML fragment from a URL and inserts it into an element.
  */
  method('load', function(uri, method) {
    // ...
  });
  window.$ = function() {
    return new _$(arguments);
  });
})();
```

Now that the API is stubbed out, it’s important to consider who might be using it and in what context. If there is an existing API that already uses the dollar function, this library will overwrite it. A simple solution is to change the name of the dollar function within the source. However, this isn’t ideal if you’re retrieving the code from an existing source-code repository; you would have to change it again each time you update the source from that repository. In that case, a better solution is to add an installer, as demonstrated here:

```javascript
Function.prototype.method = function(name, fn) {
  // ...
};
(function() {
  function _$(els) {
    // ...
  }
  _$.method('addEvent', function(type, fn) {
    // ...
  });
  window.installHelper = function(scope, interface) {
    scope[interface] = function() {
      return new _$(arguments);
    }
  };
})();
```

One possible implementation could look like this:

```javascript
installHelper(window, '$');
$('example').show();
```

Here is a more complex example that allows you to attach the functionality to a predefined namespaced object:

```javascript
// Define a namespace without overwriting it if it already exists.
window.com = window.com || {};
com.example = com.example || {};
com.example.util = com.example.util || {};
installHelper(com.example.util, 'get');
(function() {
  var get = com.example.util.get;
  get('example').addEvent('click', function(e) {
    get(this).addClass('hello');
  });
})();
```

## Using Callbacks to Retrieve Data from Chained Methods

In some cases, it isn’t a good idea to chain your methods together. For mutator methods, they’re just fine, but with accessor methods, you may wish to return the data that you are requesting, instead of returning this. Nevertheless, if chaining is your ultimate goal and you wish to keep methods consistent, you can work around this problem by using function callbacks to return your accessed data. This next example shows both of these techniques. The API class uses normal accessors (which break the chain), while the API2 class uses callback methods:

```javascript
// Accessor without function callbacks: returning requested data in accessors.
window.API = window.API || {};
API.prototype = (function() {
  var name = 'Hello world';
  // Privileged mutator method.
  setName: function(newName) {
    name = newName;
    return this;
  },
  // Privileged accessor method.
  getName: function() {
    return name;
  }
})();
// Implementation code.
var o = new API;
console.log(o.getName()); // Displays 'Hello world'.
console.log(o.setName('Meow').getName()); // Displays 'Meow'.
// Accessor with function callbacks.
window.API2 = window.API2 || {};
API2.prototype = (function() {
  var name = 'Hello world';
  // Privileged mutator method.
  setName: function(newName) {
    name = newName;
    return this;
  },
  // Privileged accessor method.
  getName: function(callback) {
    callback.call(this, name);
    return this;
  }
})();
// Implementation code.
var o2 = new API2;
o2.getName(console.log).setName('Meow').getName(console.log);
// Displays 'Hello world' and then 'Meow'.
```

## Summary

JavaScript passes all objects by reference, so you can pass these references back in every method. By returning this at the end of each method, you can create a class that is chainable. This style helps to streamline code authoring and, to a certain degree, make your code more elegant and easier to read. Often you can avoid situations where objects are redeclared several times and instead use a chain, which produces less code. If you want consistent interfaces for your classes, and you want both mutator and accessor methods to be chainable, you can use function callbacks for your accessors.

repository: luisliuchao/js-design-patterns

path: /chapter 6 - chaining.md

filename: chapter 6 - chaining.md

repo_name: luisliuchao/js-design-patterns