text:
# JupyterLab for data scientists and researchers

## Features
* Lightweight docker image with [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/), [Python 3.6](https://docs.python.org/3.6/);
* Well organised bash environment (bash history, non root user, etc.);
* *Makefile* for fast and easy everyday using;

## Docker instalation
Please follow [installation instructions](https://docs.docker.com/install/) from the official site.

## Using
Prebuilt image is available from the [docker hub](https://hub.docker.com/r/dbaibak/jupyter-lab/)

### [With makefile](#makefile):
* `make pull` - pull latest image;
* `make run local=path_to_the_project_folder name=name_of_the_container` - build a new container and mount local project folder. The container is available by the url http://<IP_ADDRESS>:8888/;
* `make start name_of_the_container` - start specific container
* `make stop name_of_the_container` - stop specific container
* `make stop_all` - stop all containers
* `make remove name_of_the_container` - remove specific container
* `make bash name_of_the_container` - run bash for specific container

## Examples

### [Workflow](#workflow)
* `make pull` - pull image from the docker hub;
* `make run local=$PWD name=my-lab` - create container with name *my-lab* and mount it to the *current directory*;
* got to the url http://<IP_ADDRESS>:8888/ - use JupyterLab;
* `make bash my-lab` - if you need access to the bash;
* `pip install --user matplotlib` - install new package;
* `make stop my-lab` - stop container;
* `make start my-lab` - start container;

### Build container with specific enviroment
[Dockerfile](https://github.com/DanilBaibak/docker-jupyter-lab/blob/master/examples/ml_env/Dockerfile) that installs a list of the specific requirements. Can be useful if you are using few containers with the same list of the libraries.

repository: vinayak-pathak/docker-jupyter-lab

repo_name: vinayak-pathak/docker-jupyter-lab


text:
## Envelop

Envelop is a lightweight JavaScript library for wrapping GraphQL execution layer and flow, allowing developers to develop, share and collaborate on GraphQL-related plugins, while filling the missing pieces in GraphQL implementations.

`@envelop/core`: [![npm version](https://badge.fury.io/js/%40envelop%2Fcore.svg)](https://badge.fury.io/js/%40envelop%2Fcore)

> Envelop is created and maintained by [The Guild](https://the-guild.dev/), and used in production by our clients.

### In depth

Our goal is to allow developers keep the original GraphQL interfaces, while adding plugins to enrich the feature set of you application quickly, while sharing ideas, code and plugins with other developers.

Envelop is agnostic to the HTTP server you use, so it's not a traditional server. We do not aim to provide a complete server, you can use Envelop with any environment (NodeJS or browser) and any type of GraphQL workflow (client / server, client-side execute, or server to server). So any piece of code that uses GraphQL's `execute` can benfit from that layer.

The core of Envelop is zero-dependency, and will only apply changes to your GraphQL execution based on the plugins you wish to use. It can be integrated with any GraphQL server that follows [the execution phase, as defined in the GraphQL spec](https://spec.graphql.org/June2018/#sec-Executing-Requests) and let you provide your own lifecycle methods.

Separating the execution workflow and the logic that it runs in each phase allow you to write reusable piece of code, like logging, metric collection, error handling, custom validations, resolvers tracing (and integration with any external), authentication, authorization and much more, without the needs to write it explictly every time, or in any project or microservice. With Envelop, you can easily create a wrapper around your common logics, and share it with others.

## Getting Started

Start by adding the core of Envelop to your codebase:

```
yarn add graphql @envelop/core
```

Then, create a simple Envelop based on your GraphQL schema:

```ts
import { envelop, useSchema } from '@envelop/core';

const mySchema = buildSchema( ... ); // GraphQLSchema

const getEnveloped = envelop({
  plugins: [
    useSchema(mySchema)
  ],
});
```

The result of `envelop` is a function, that allow you to get everything you need for the GraphQL execution: `parse`, `validate`, `contextBuilder` and `execute`. Use that to run the client's GraphQL queries. Here's a psuedo-code of how it should looks like:

```ts
const httpServer = createServer();

httpServer.on('request', (req, res) => {
  // Here you get the alternative methods that are bundled with your plugins
  const { parse, validate, contextFactory, execute, schema } = getEnveloped();

  // Parse the initial request and validate it
  const { query, variables } = JSON.parse(req.payload);
  const document = parse(query);
  const validationErrors = validate(schema, document);

  if (validationErrors.length > 0) {
    return res.end(JSON.stringify({ errors: validationErrors }));
  }

  // Build the context and execute
  const context = await contextFactory(req);
  const result = await execute({
    document,
    schema,
    variableValues: variables,
    contextValue: context,
  });

  // Send the response
  res.end(JSON.stringify(result));
});

httpServer.listen(3000);
```

Behind the scenes, this simple workflow allow you to use **Envelop plugins** and hook into the entire request handling flow.

Here's a simple example for collecting metrics and log all incoming requests, using the built-in plugins:

```ts
const getEnveloped = envelop({
  plugins: [useSchema(schema), useLogger(), useTiming()],
});
```

> Envelop `plugins` are based on a simple event-based contact, that allow you easily to add more logic to your app. You can easily share and collaborate on plugins that you find generic.

## Integrations / Examples

Envelop provides a low-level API at consumpion of the output, but a rich API while using it with plugins. Based on that, it's possible to integrate Envelop with many tools.

We recommend on using [`graphql-helix`](https://github.com/contrawork/graphql-helix) as request pipeline orchestrator, as it allows the maximum flexibility and you can easily override/manage every part of the pipeline with Envelop.

Here's a list of integrations and examples:

| Server/Framework | Fully supported? | Example                                     |
| ---------------- | ---------------- | ------------------------------------------- |
| Node's `http`    | V                | [`basic-http`](./examples/simple-http)      |
| GraphQL-Helix    | V                | [`graphql-helix`](./examples/graphql-helix) |
| Apollo-Server    | Almost           | [`apollo-server`](./examples/apollo-server) |

> Since Envelop is not a HTTP server, and just a wrapper around GraphQL request pipeline - it's possible to integrate it with any server/framework, if it's flexible enough and allow you to specify the pipeline methods\*.

## Available Plugins

We provide a few built-in plugins within the `@envelop/core`, and many more plugins as standalone packages.

| Name                 | Package                                                                | Description                                                                    |
| -------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| useSchema            | [`@envelop/core`](./packages/core#useschema)                           | Simplest plugin to provide your GraphQL schema.                                |
| useErrorHandler      | [`@envelop/core`](./packages/core#useerrorhandler)                     | Get notified when any execution error occurs.                                  |
| useExtendContext     | [`@envelop/core`](./packages/core#useextendcontext)                    | Extend execution context based on your needs.                                  |
| useLogger            | [`@envelop/core`](./packages/core#uselogger)                           | Simple, yet powerful logging for GraphQL execution.                            |
| usePayloadFormatter  | [`@envelop/core`](./packages/core#usepayloadformatter)                 | Format, clean and customize execution result.                                  |
| useTiming            | [`@envelop/core`](./packages/core#usetiming)                           | Simple timing/tracing mechanism for your execution.                            |
| useGraphQLJit        | [`@envelop/graphql-jit`](./packages/plugins/graphql-jit)               | Custom executor based on GraphQL-JIT.                                          |
| useParserCache       | [`@envelop/parser-cache`](./packages/plugins/parser-cache)             | Simple LRU for caching `parse` results.                                        |
| useValidationCache   | [`@envelop/validation-cache`](./packages/plugins/validation-cache)     | Simple LRU for caching `validate` results.                                     |
| useDepthLimit        | [`@envelop/depth-limit`](./packages/plugins/depth-limit)               | Limits the depth of your GraphQL selection sets.                               |
| useDataLoader        | [`@envelop/dataloader`](./packages/plugins/dataloader)                 | Simply injects a DataLoader instance into your context.                        |
| useApolloTracing     | [`@envelop/apollo-tracing`](./packages/plugins/apollo-tracing)         | Integrates timing with Apollo-Tracing format (for GraphQL Playground)          |
| useSentry            | [`@envelop/sentry`](./packages/plugins/sentry)                         | Tracks performance, timing and errors and reports it to Sentry.                |
| useOpenTelemetry     | [`@envelop/opentelemetry`](./packages/plugins/opentelemetry)           | Tracks performance, timing and errors and reports in OpenTelemetry structure.  |
| useGenericAuth       | [`@envelop/generic-auth`](./packages/plugins/generic-auth)             | Super flexible authentication, also supports `@auth` directive .               |
| useAuth0             | [`@envelop/auth0`](./packages/plugins/auth0)                           | Validates Auth0 JWT tokens and injects the authenticated user to your context. |
| useGraphQLModules    | [`@envelop/graphql-modules`](./packages/plugins/graphql-modules)       | Integrates the execution lifecycle of GraphQL-Modules.                         |
| useGraphQLMiddleware | [`@envelop/graphql-middleware`](./packages/plugins/graphql-middleware) | Integrates middlewares written for `graphql-middleware`                        |
| useRateLimiter | [`@envelop/rate-limiter`](./packages/plugins/rate-limiter) | Loimoit request rate via `@rateLimit` diractive                        |

## Sharing `envelop`s

After an `envelop` has been created, you can share it with others as a complete layer of plugins. This is useful if you wish to create a predefined layer of plugins, and share it with others. You can use the as a shell and as a base for writing sharable pieces of servers.

Here's a small exmple for sharing envelops:

```ts
// Somewhere where you wish to create the basics of what you wish to share
// This defined the base plugins you wish to use as base.
const myBaseEnvelop = envelop({
  plugins: [useOrgAuth(), useOrgTracing(), useOrgLogsCollector()],
});

// Later, when you create your own Envelop, you can extend that and add custom plugins.
// You can also specify the schema only at this point
const myEnvelop = envelop({
  extends: [myBaseEnvelop],
  plugins: [useSchema(myServerSchema), useMyCustomPlugin()],
});
```

This approach allow developers to create a base Envelop and share it across the organization: you can define your monitoring setup, logging, authentication, etc, only once in a common package, and share it with others without losing the ability to extend it.

## Write your own plugin!

Envelop plugins are just object with functions, that provides contextual implementation for before/after of each phase, with a flexible API.

Here's a simple example that allow you print the execution params:

```ts
const myPlugin = {
  onExecute({ args }) {
    console.log('Execution started!', { args });

    return {
      onExecuteDone: ({ result }) => {
        console.log('Execution done!', { result });
      },
    };
  },
};

const getEnveloped = envelop({
  plugins: [
    /// ... other plugins ...,
    myPlugin,
  ],
});
```

> Feel free to share you plugin with us, or with the community. Sharing = Caring!

#### The plugin interface

You can find it here: https://github.com/dotansimha/envelop/blob/main/packages/types/src/index.ts#L50

#### Execution Lifecycle

By extending the GraphQL execution pipeline, we allow developers to write reusable plugins, that can be shared with others easily, as NPM packages. So instead of delivering a bloated GraphQL server with tons of features, we allow you to choose the HTTP server you prefer, the request pipeline you prefer, and the feautures you prefer.

We wrap the execution pipeline of GraphQL operation, and allow Envelop plugins to do the following:

- `parse`
  - Hook into the before/after of this phase
  - Overide the parse function
  - Access to the parsed result
  - Modify the parsed result
- `validate`
  - Hook into the before/after of this phase
  - Overide the validation function
  - Access to the validation error results
  - Modify the validation results
  - Add custom validation rules
- `contextFactory`
  - Hook into the before/after of this phase
  - Access to the initial HTTP request context
  - Extend the context with custom data
  - Replace the context object
- `execute`
  - Hook into the before/after of this phase
  - Extend the execution context
  - Access to all execution parameters
  - Replace the execute function
  - Access to the results / error of the execution
  - Access to before / after resolvers calls
  - Extend resolvers behaviour
  - Access resolvers parameters
  - Replace / modify the execution result
- `subscribe`
  - Hook into the before/after of this phase
  - Extend the execution context
  - Access to all execution parameters
  - Replace the execute function
  - Access to the results / error of the execution
  - Access to before / after resolvers calls
  - Extend resolvers behaviour
  - Access resolvers parameters
  - Replace / modify the subscription result

We also allow you to change the GraphQL schema during execution - so if your server has a schema that could change dynamically, you can always update it. As a result, we trigger `schemaChange` event that allow plugins respond accordingly.

### Contributing

If this is your first time contributing to this project, please do read our [Contributor Workflow Guide](https://github.com/the-guild-org/Stack/blob/master/CONTRIBUTING.md) before you get started off.

Feel free to open issues and pull requests. We're always welcome support from the community.

For a contribution guide specific to this project, please refer to: http://graphql-code-generator.com/docs/custom-codegen/contributing

### Code of Conduct

Help us keep GraphQL Codegenerator open and inclusive. Please read and follow our [Code of Conduct](https://github.com/the-guild-org/Stack/blob/master/CODE_OF_CONDUCT.md) as adopted from [Contributor Covenant](https://www.contributor-covenant.org/)

### License

[![GitHub license](https://img.shields.io/badge/license-MIT-lightgrey.svg?maxAge=2592000)](https://raw.githubusercontent.com/apollostack/apollo-ios/master/LICENSE)

MIT

license: MIT

repository: tvvignesh/envelop

branch_name: refs/heads/main

extension: md

gha_license_id: MIT

license_type: permissive

repo_name: tvvignesh/envelop


text:
# FVTT-DD-Importer
Allows Importing DungeonDraft map files into FoundryVTT

**Version 0.2**: Imports Walls, Doors, and Lights. No cave walls yet.

The import dialog is accessible by right clicking on a scene in the sidebar.

Open your Dungeon Draft map file in any text editor and paste the contents into the dialog.

Manifest: `https://raw.githubusercontent.com/moo-man/FVTT-DD-Import/master/module.json`

branch_name: refs/heads/master


text:
# Redirect to canonical domain name/scheme

Provides framework agnostic redirects to canonical host/scheme
without ```.htaccess```.

Example:

```php
$canonical = new \cronfy\canonical\Request();
$canonical->setCanonicalHost('example.com');
$canonical->setCanonicalScheme('https');
$canonical->redirectToCanonical();
```

If requested host/scheme does not match canonical ones, user will be redirected:

```
http://www.example.com/contacts/ => https://example.com/contacts/
```

## Installation

```bash
composer require cronfy/canonical dev-master
```

## Related

>> SO [Как сделать всё и сразу в mod_rewrite?](https://ru.stackoverflow.com/questions/542869/%D0%9A%D0%B0%D0%BA-%D1%81%D0%B4%D0%B5%D0%BB%D0%B0%D1%82%D1%8C-%D0%B2%D1%81%D1%91-%D0%B8-%D1%81%D1%80%D0%B0%D0%B7%D1%83-%D0%B2-mod-rewrite)

> [Вы идёте неправильным путём](https://ru.stackoverflow.com/a/542870/200260)

repository: cronfy/canonical

repo_name: cronfy/canonical


text:
# Sneakers Archive

Project to obtain images of sneakers from different brands using web scraping.
Made for educational purposes.

## Installation

Sneaker Archive requires Python 3.6 or higher and pip installed on your computer.
Poetry is optional to generate the virtual environment.

To start open a terminal and clone the project.

```
git clone https://github.com/rsolanoweb/sneakers-archive.git
```

Once the project has been cloned, access the directory.

```
cd sneakers-archive
```

Install the necessary packages or install the virtual environment with poetry.

```
pip install -r requirements.txt
```
or
```
poetry install
```

Now is the time to run the program and start getting the images you want.

```
python main.py
```
or
```
poetry run python main.py
```

## Tests

Check the correct operation of the program.

```
pytest .
```
or
```
poetry run pytest .
```

repository: rsolanodev/sneakers-archive

repo_name: rsolanodev/sneakers-archive


text:
# http-dump

Logs http headers to stdout. Can be run as a docker container from `scalify/http-dump`.

## usage

    docker run --rm \
        -p "8080:8080" \
        -it scalify/http-dump

## License

    MIT License
    Copyright (c) 2017 Scalify GmbH

repository: aklinkert/http-dump

repo_name: aklinkert/http-dump


text:
---
aliases:
- /2020/12/11/Naughty-by-Numbers
date: 2020-12-11 16:25:13
description: A snarky interactive lesson on classification and surveillance
hide: false
image: images/naughty2.png
layout: post
search_exclude: false
title: 'Naughty by Numbers: Classifications at Christmas'
toc: false
categories:
- classification
- foundations

---

## Head over to [https://hedges.belmont.edu/~shawley/naughty/](https://hedges.belmont.edu/~shawley/naughty/) to see this post.

(It uses enough JavaScript that this blogging platform will mangle it.)

![preview image](https://hedges.belmont.edu/naughty/naughty2.png)

repository: drscotthawley/blog

gha_language: Jupyter Notebook

repo_name: drscotthawley/blog


text:
---
company: Burberry
role: Lead Developer
from_date: 2019-01
to_date: 2020-01
location: London, UK
technologies:
  - Typescript
  - React
  - Redux
  - redux-saga
  - Next.js
  - Gatsby
  - JAMstack
  - SASS
  - Styled components
  - Backpack
  - style-dictionary
  - Docker
  - Terraform
  - Cypress
  - GitLab
  - GitLab CI
  - AWS
---

I joined to help shape the new micro front end architecture, part of the wider re-platforming of Burberry’s ecom. During my contract I led two of our distributed teams of developers, located in Minsk and Shanghai, while working with stakeholders and designers in-house. Day to day activities with the team included working on stories, mentoring other team members, reviewing merge requests, working with the BA to review stories and prepare sprint goals and working with the designers to make sure all the edge cases were covered.

Particular focus was also put on working with across-teams to agree on high level solutions and create a consistent development workflow. Consistency and simplicity were key, to ensure that the cognitive load was kept to a minimum and the code produced would follow our standards regardless of where it was produced.

As part of my assignment, I also worked closely with the design team to help build the company’s design system: from defining the key strategy and technologies to use to building a set of design tokens to be used across web, email and native apps.

branch_name: refs/heads/dev

gha_language: TypeScript


text:
# Cumulocity LoRa plugin

This is the Cumulocity LoRa plugin.

It is intended to be installed in devicemanagement application only.

This plugin requires the microservice feature enabled on your tenant to work. If you're on a trial tenant please contact your sales contact to do so, or contact support on Empower.

Once microservice feature is enabled, please install the github-proxy microservice that can be downloaded from this URL: https://github.com/SoftwareAG/cumulocity-lora/releases/download/v0.30/github-proxy.zip

When the github-proxy microservice is installed, just go to devicemanagement application, then to lora/config menu.
From there just select the microservices you want to install on your tenant.

You'll see 2 kinds of microservices: LNS connectors, usually named lora-ns-, and codecs, usually named lora-codec-

![](assets/img/config.png)

## User documentation

### Adding a new LNS connector

In devicemanagement application go to menu LoRa/LoRa network servers.
Simply give a name to your connector, then select the type of LNS you want to connect to (the type you're looking for will only show up if the corresponding microservice is installed).

![](assets/img/lora_ns.png)

Ensure that le LNS you want to connect to Cumulocity is properly configured (some LNS require the creation of an API key for instance).
Once the type is selected, simply follow the wizard instructions.
Already provisioned devices will be automatically added to Cumulocity once they send data and their data are forwarded by the LNS to Cumulocity.

Example for TTN:

![](assets/img/lora_ns_TTN.png)

And another for Chirpstack:

![](assets/img/lora_ns_Chirpstack.png)

### Automatic device import

Once the connector is created, Cumulocity will automatically create an HTTP route in the LNS to push data from the LNS to the current Cumulocity tenant.
Moreover, all existing devices in the LNS will be automatically created in Cumulocity once they send data and data are pushed to Cumulocity through the configured HTTP route. Those devices will be created without a codec associated to them, you will therefore need to go to the device detailed view in order to choose the right codec.
Any payload sent to Cumulocity priori codec selection won't be lost: they will be stored and you will be able to process them once you have chosen the right codec.

![](assets/img/lora_device_codec_selection.png)

### Provisioning new LoRa devices

In devicemanagement-lora application go to menu LoRa/LoRa devices.

![](assets/img/lora_devices.png)

In order to provision the device in an LNS, you need to check the box labeled "Provision device in LoRa network server": you will then be able to choose the LNS connector related to the LNS you want to provision the device into.
Depending on the type of LNS connector you choose, different fields will appear to finalise the device provisioning in that specific LNS.

Example for TTN:

![](assets/img/lora_devices_TTN.png)

And another for Chirpstack:

![](assets/img/lora_devices_Chirpstack.png)

### Provisioning new gateways

In devicemanagement-lora application go to menu LoRa/LoRa gateways.

![](assets/img/lora_gateways.png)

Depending on the LNS connector you will choose, different fields will appear.

Example with TTN:

![](assets/img/lora_gateways_TTN.png)

Another with Actility:

![](assets/img/lora_gateways_Actility.png)

In order to be able to provision a gateway, some LNS require specific authorization. For example TTN requires a user API key, as an application API key will only allow you to manage devices in a specific application.

### Device management

LoRa devices, identified by a JSON fragment `lora_ns_device_LoRaDevice`, can be managed from a specific view accessible through the submenu "LoRa" on the device detailed view.

![](assets/img/lora_device_view.png)

Several actions can be performed in this view:

1. Configure device codec

You can change the codec at any moment. If you select the wrong and it failed to process an incoming payload, the payload will stay in an unprocessed state and you will be able to reprocess it later.

2. Codec options

You can enable debug mode, which will generate a special event for each incoming payload that will contain the full decoded payload.

You can also choose to store the last measurements on the device itself, so you will be able to use them on specific widgets but also to add them as columns in the "all devices" table.

3. Process unprocessed payloads

Devices that are automatically imported won't have a codec configured, and payloads will be stored in an unprocessed state.
You will be able to process them once you have configured a codec.

4. Switch LNS

You can switch a device to another LNS. When doing so the device will be deprovisioned from its current LNS and provisioned to the newly selected LNS.

5. Send commands

Depending on the codec configured on the device, various commands will be available.
Most device will have a command to retrieve their current configuration. When such commands are processed by the device, an uplink containing the device configuration will be received and the configuration will be stored on the device in Cumulocity as a JSON document available in the configuration tab of the device:

![](assets/img/lora_device_configuration.png)

You can either update the device configuration by modifying the JSON directly or by sending a command from the LoRa tab. Commands available on the LoRa tab will have their input fields prefilled with the values stored in the device configuration:

![](assets/img/lora_device_send_command.png)

When a command is sent it will go through the usual Cumulocity operation lifecycle:

PENDING -> EXECUTING -> SUCCESS or FAILED

The status of the operation will be updated when the LNS send a downlink update event.

Note that not all LNS support downlink update event or that it as to be switched on explicitely either on the device profile or on the LNS configuration.

### LNS events

All LNS events will be available as events on the LNS connector managed object in their raw JSON format as sent by the LNS.

![](assets/img/lora_raw_LNS_uplink.png)

Uplink events will also be available on the devices in a form that contains the fport, the payload in hex format, whether the payload was processed or not and other relevant information.

![](assets/img/lora_device_raw_payload.png)

### Codec IDE

The framework offers the ability to quickly build codec directly from Cumulocity UI with an integrated IDE based on Monaco Editor.

To enable this feature you need to install the custom codec microservice `lora-ns-custom`.

The IDE is accessible on LoRa/Custom codecs menu and is devided in 3 parts:

- Uplink processing (decoder)

![](assets/img/custom_codecs_decoder.png)

- Downlink processing (encoder)

![](assets/img/custom_codecs_encoder.png)

- Operations definition

![](assets/img/custom_codecs_operations.png)

Once you're happy with your codec and you saved it, it will be accessible from any LoRa device Lora tab as a model of the custom codec.

repository: SoftwareAG/cumulocity-lora

gha_language: Java

repo_name: SoftwareAG/cumulocity-lora


text:
# Introduction

## Who is this article for?

In this article, I will show you how I built **a full stack architecture using PHP, Symfony, Doctrine, GraphQL, ApolloJS and React**. If you are considering to set up an architecture with GraphQL in PHP, I encourage you to read this article because I talk about each component and interaction between them.

I also take some time to show how I solved the most pressing challenges I found while integrating these technologies.

The code we are going to explore is an open source App built with this architecture. When I was studying, I felt that there were missing code examples on the community, so **I hope that the repositories we are using will be usefull to your own study**.

Here they are:

- [PHP backend with Symfony, Doctrine and Overblog GraphQL](https://gitlab.com/bruno.p.reis/nosso-jardim)

- [JS frontend with Apollo Client, React Apollo, React and Semantic UI React](https://gitlab.com/bruno.p.reis/nosso-jardim-client)

As they say, **a repo is worth a thousand words**!

Don't take the example here as "Best Practices". It's impossible to talk about "Best Practices" in a so fast evolving technology scenario, especially for GraphQL based libs, since it was released two year ago.

So please do as you were eating a fish, keep the meat and spit the fishbone.

## Driving Forces

Please, allow me a more abstract and high level talk before getting into the code.

React is a very nice frontend technology and has some paradigm shifts for a frontend tech. **It's beauty lies in the fact that you always have a predictable view for a specific model state.** The hadaches with a lot of binds and listeneres are mostly gone and the code is a lot more clean and therefore maintainable.

Talking in other terms, a React frontend is a pure function where you pass a model and get a view. Together with Redux, that has a similar approach for state management, they make a very cool tool for the frontend apps, but something was missing.

**The R & R couple (React and Redux) still lack a good way to integrate with assynchronous data**. So, I started a quest on what to use there. I researched a lot of redux patterns and libs to do this job, but they all seemed to require to much to do the job.

Looking for solutions, I was hearing a lot of buzz about Relay, and I decided to give it a try. Specially because I like the facebook policy of releasing code they actually use. So I started studying relay.

Relay docs were too cryptic to me and I gave up on it on my first try. A little later, I met Apollo, backed by the meteor guys. I found they had a better documentation and community, so I decided to give it a try.

I started my first app using a GraphQL client. Using it **I discovered that Apollo was able to do a lot more than just fetching data from the server. It's cache layer is very well managed, it is able to normalize and denormalize data into there, it can manage subscriptions and also can agregate queries to save requests.**

Using it I was able to mantain a very nice and organised codebase, backed by a very powerfull tool to integrate assynchronous data with my view. **To me, the declarative way of composing the views with HOCs wrapping the views fitted like a glove in my hands.**

To use Apollo, I had to learn **GraphQL, that is a superb language and technology to expose and query an API**. Created inside Facebook and used into real life hard core projects in an intensive way before being released, GraphQL really improves a lot of things over old rest communication. ( Don't take me for granted. [Take a look here.](http://graphql.org/learn/)

I'm already a **PHP/Symfony** developer for a long time, so that was my choice for backend. I confess I was very inclined to work with the JS implementation at first, but adding too much new stuff to a new project is never good in my experience.

I found mature libs in PHP and the app organization, after a lot of refining, came out to be very good and clean in the backend too. Indeed, **GraphQL helped me a lot on that Backend organisation.**

Another decision taken was running most tests over the API layer, what also proved to be a very good decision. I like a lot **TDD**, but finding the correct amount of tests to move fast and also finding the correct layer to test is a fine art to me.

Well, this stuff is hard to talk about, so please take a look at the code([1](https://gitlab.com/bruno.p.reis/nosso-jardim)[2](https://gitlab.com/bruno.p.reis/nosso-jardim-client)) to get an overall feeling befor we start. Or, if you are not in a mood, just follow along.

## Our Roadmap

    Introduction
        Who is this article for?
        Driving Forces
        Our Roadmap (<-you are here)
    Backend
        The App we are gonna look at
        Installation
        The API Schema
        Overblog GraphQL
        The Schema Declaration - The entrance to the backend
        Following a complete backend query
        A little Recap
        Cors
        Error Handling
        Testing your code
    Frontend
        Installing the frontend
        Overview
        Our next destination: MessageStatus View Component
        Integrating data with containers
        Diving deepen into the PEERS query
        Integration between Apollo containers and also Redux
        Verification Point

# Backend

On the backend side, we are going to explore a [PHP backend with Symfony, Doctrine and Overblog GraphQL](https://gitlab.com/bruno.p.reis/nosso-jardim)

## The App we are gonna look at.

The view is usually the easiest part to understand an application domain. So let's do it before we install the backend and look at it's API.

We are gonna work on a **Knowledge Management App**. The main purpose of this app is to organize knowledge that is shared through Telegram and other channels in the future.

The mais screen, as shown below, is where you organize messages in specific threads or conversations (Conversas in Brazilian Portuguese) and put tags on those threads.

![Create a Thread and Tag it!](./images/createThread.gif)

This App gives us interesting data structures to serve us as an example:

1. A paginated list - Messages
2. A non paginated list - Threads
3. A tree - Subtopics (not shown in the gif)
4. Lots of simple views, menus and buttons.
5. Forms and data edition - Add Tag, Edit Subtopic, Add Subtopic, and so on...

**These examples are all available on the repository code** and can ilustrate how to handle these data structures on both front and backend.

Ok, now that you understand what we will be working on, you might want to grab the backend code.

## Installation

1. [Clone the backend repo.](https://gitlab.com/bruno.p.reis/nosso-jardim)
2. Run composer install and configure your parameters
3. Import Fixtures and Data
4. Start the php server
5. Read the next section so that you can look at GraphiQL knowing the app

## The API Schema

You can run [queries](https://dev-blog.apollodata.com/the-anatomy-of-a-graphql-query-6dffa9e9e747) using GraphiQL and see the results right there, with the docs on the right where you can check for the types and formats.

![Running Queries](./images/runningQueries.gif)
![Looking at the Documentation](./images/documentation.gif)

Go ahead, take some time now, navigate through GraphiQL and look at the schema and queries available to understand our domain. I should be under /graphiql in the address your php server is runnig. Probably on <IP_ADDRESS>:8000/graphiql

## Overblog GraphQL

To build the GraphQL server over symfony, we are using the overblog/GraphQLBundle lib. (https://github.com/overblog/GraphQLBundle)

It's a very good lib that integrate straight into symfony creating the GraphQL endpoint and adding nice features to the Webonys lib. One very special feature it has is the [Expression Language|https://github.com/overblog/GraphQLBundle/blob/master/Resources/doc/definitions/expression-language.md]

As you can see on it's [requirements|https://github.com/overblog/GraphQLBundle/blob/master/composer.json], it is built over cool libs. I want to especially note these two:

	1 - overblog/graphql-php-generator - This is responsible for reading a nice and clean yml and convert it into the GraphQL type objects. It will all happen under the hood and all you will need to touch is the yml and the resolvers.

	2 - webonyx/graphql-php - This is the real engine of our car. A PHP port of GraphQL reference implementation. Very stable and ready to use. Please take a look at the docs and I call special attention to the

## The Schema Declaration - The entrance to the backend

Our schema declaration is under src/AppBundle/Resources/config/graphql/
Queries are defined in the Query.types.yml. The fields there are the system queries, possible args are defined there and the resolvers are also set there.

So we can understand that this file is the entrance on our system. It defines the API interface with the outter word.

## Following a complete backend query

So, as we have seen, the schema is the entrance to the backend. From there, it direct the calls to specific resolvers, using the nice [expression language](https://github.com/overblog/GraphQLBundle/blob/master/Resources/doc/definitions/expression-language.md) ofered by the overblog bundle.

Take a look at the item field or query:

```yml
#Queries.types.yml
#...
        fields:
            item:
                type: "Item"
                args:
                    id:
                        type: "ID!"
                resolve: "@=service('app.resolver.items').findOne(args)"
#...
```

It defines the id argument, set it's type as ID and also inform that it's required (!). It also declares the resolver as bing the service defined as "app.resolver.items", that is then requested from symfony dependency injection container. It also passes the received args object straight to the resolver method "findOne". Let's look at "findOne":

```php
#AppBundle\Resolver\ItemsResolver.php
#...
	public function findOne($args)
    {
        $id = $args['id'];
        $item = $this->repo()->find( $id );
        if(!$item) throw new UserErrorException("Item not found for id " . $id, 1);
        return $item;
    }
#...
```

That's a normal code you would expect in a controller, but as you can see, the GraphQL bundle adds some very nice goodies, saving our time on repetitive tasks we usually need to do on controllers or to build as a layer by ourselves.

First, it defines the expected input. Before passing data to the resolver it will check for the 'id' field requirement and type. So, you can be sure it's there when you write your code.

Second, it will transform the object we return in the desired response. I say desired response because this query can be called in different ways, depending on the client needs and will return different fields and even "join fields" according to to what the client has requested.

After building the response on the server, what is done by the resolver, the GraphQL layer there will check to see if the values are correct according to the return type of that query.

In that case, the declared type for that query is "Item". And, being so, it will look at Item.types.yml to understand that type and it's field declarations. If the response field types are ok, validated against that, then it will send that response to the client as a 'JsonResponse'.

```yml
Item:
    type: object
    config:
        description: "A media type"
        fields:
            id:
                type: "ID!"
                description: "The id of the item."
            subtopic:
                type: "Subtopic"
                description: "The subtopic refined by this item"
            name:
                type: "String"
                description: 'The name of this item'
```

If you are like me and like to take a look at the source to understand what you are using, you can start looking at [the controller endpoint action](https://github.com/overblog/GraphQLBundle/blob/master/Controller/GraphController.php#L20). That's the controller that establishes the graphql endpoint for all the requests. From there it will call the executor and make all this fun to happen. And from there you can explore a little more how the bundle integrate the webonyx graphql lib into symfony.

## A little Recap

Well, this is a lot of information so far, so I suggest a little recap now. I know I explained a lot about GraphQL and it's inner workings, and also about the libs we are using. So, I think that it's important to us to see, after setting up all this army, all this power, what was really left to us:

1 - define a single field entry on the Query.types.yml
2 - writting the resolver, completely focused on my business logic.
3 - nothing else.

Hey, this is beautifull! Take a look:
    - no need to write a lot of controllers
    - no need to think about routes since you have only one endpoint
        - at least to me thinking in terms of fields and objects is a lot easier than comming up with some restful logic to map uris to business
    - good documentation being generated
        - most part of it by introspection
    - a tool to interact and test your api (graphiQL).
    - The best: focus on your business logic.

As I've said before, while building the backend my first objective was only to have a clean and nice schema to be accessed from outside. But, I got surprised on how the GraphQL approach made my write less boilerplate code on the server. And also satisfied by the resulting code organization.

Let's talk a little now about some other issues I had to deal in the server development.

## Cors

This app is not on a prod server yet, but I intend to keep the server in a different env, so I installed the [NelmioCorsBundle|https://github.com/nelmio/NelmioCorsBundle]

The configurations today are very open and will need to be a lot more string on a prod server. But, I just wanted you to note that it's running and will help you to avoid a lot of errors seen on the frontend client.

It's also worth noticing that I had to add 'content-type' as an allowed header in it's config.

If you don't know this bundle yet, it's worth taking a look at it. It will manage the headers sent and specially the OPTIONS pre-flight requests to enable cross origin resource sharing. In other words, will enable you to call your API from a different domain.

## Error Handling

Error handling is a very open topic on GraphQL world. The specs don't say a lot ([1](https://facebook.github.io/graphql/#sec-Errors),[2](https://facebook.github.io/graphql/#sec-Executing-Operations)) and are open to a lot of interpretations. And, as we can see by the community, there are a lot of different opinions on how to handle them ([1](https://voice.kadira.io/masking-graphql-errors-b1b9f15900c1),[2](https://medium.com/@tarkus/validation-and-user-errors-in-graphql-mutations-39ca79cd00bf))

Being GraphQL so recent, it's expected that you don't have well established best practices on it yet. And this is something nice to take into consideration when designing your system. Maybe you see a better way of doing things. So, please do it, test, and share with us.

Overblog deals with errors in a manner that is good enough me. First, it will add normal validation errors when it encounter them on the data validation of input or output types.

Second, it will handle the exceptions thrown in the resolvers. When it catches an exception, it will add an error to the response. Almost all exceptions are added as an "internal server error" generic message. The only two Exception types (and subtypes) that are not translated to this generic message are:

- ErrorHandler::DEFAULT_USER_WARNING_CLASS
- ErrorHandler::DEFAULT_USER_ERROR_CLASS

These can be [configured](https://github.com/overblog/GraphQLBundle/blob/master/DependencyInjection/Configuration.php#L101) on your config.yml with your own exceptions:

```yml
overblog_graphql:
    definitions:
        exceptions:
            types:
                errors: "\\AppBundle\\Exceptions\\UserErrorException"
```

So in the code we have seen before:

```php
		#AppBundle\Resolver\ItemsResolver.php
		...
		public function findOne($args)
	    {
	        $id = $args['id'];
	        $item = $this->repo()->find( $id );
	        if(!$item) throw new UserErrorException("Item not found for id " . $id, 1);
	        return $item;
	    }
        ...
```

if you pass a non existent id to that query, it will return a nice user friendly message:

[Sreen Capture of this happening on GraphiQL]

To understand it a little deeper, please put your diving mask and look at the class that implements this handling: "Overblog\GraphQLBundle\Error\ErrorHandler"

The ErrorHandles is also responsible to put the stack trace on the response. But, it will only do it if the symfony debug is turned on. That is normally done on the "$kernel = new AppKernel('dev', true);" call. I encourage you to test sending a non existent id to that query with debug==true and with debug==false and seing the response. You can do that on GraphiQL.

To finalize our explanation on this subject, it's also worth noticing that exceptions are also logged on dev.log.

## Testing your code

The most radical XP coders would say that legacy code is any code that is not tested. Even if it was written today. I'm not so hard on that, but I really like to have a good test suite over my apps. In fact, we started this app with a very different architecture, using other libs and layers, and we would probably not have being able to refactor it to it's actual state of art without it's test suite.

I did not only refactor the app, but also refactored a lot the test suite itself. It now has a nice utility belt, with tools that enable a good test to be written in the top of the API layer, with calls being made to the GraphQL api.

To speed up a little, these calls are not simulated as being http calls, but are indeed calls to the GraphQL processor. Given that we have only one endpoint, it does not make sense to test it everytime.

The response of those calls will allways be a json like structure. A graph of objects, represented and validated by specific GraphQL types. So, I decided to put [peekmo/jsonpath|https://github.com/Peekmo/JsonPath] to work to be able to query those responses in a cleaner syntax.

At least to me querying for:

```php
'0.subtopics.0.id'
```

is easier than for:

```php
$data[0]['subtopics'][0]['id']
```

And readability is a topic that I like to specially enforce on test suites.

After fighting with it without real arguments for years and years, I now admit without any pain or hurt that PHP code looses a lot to other popular languages in the readability aspect of it's syntax. So, any help on that is welcome.

Let's take a look at a complete test. Comments are on the code to help your understanding:

```php

	function helper() {
        return new SubtopicsTestHelper($this);
    }

	/** @test */
	public function shouldDeleteASubtopic()
	{
		// main responsible for executing queries against graphql layer
		// return a function that enable a json path query on the results
	    $h = $this->helper();

	    //register two subtopics in the first level (children of root node)
	    // ['name'=>'sub1'] is the variables part of the query
	    $h->SUBTOPICS_REGISTER_FIRST_LEVEL(['name'=>'sub1']);
	    $h->SUBTOPICS_REGISTER_FIRST_LEVEL(['name'=>'sub2']);

	    // this calls for the root subtopics query. It's executed on the first call that ends just after (['root']=>true)
	    // that will respond with another function that will enable querying the resonse
	    // that function is called with two arguments:  '0.subtopics', '0.subtopics.0.id'
	    // and it returns an array with two results, one for each argument, that are processed by list and assigned to $subtopics and $subId_1
	    list(
	        $subtopics,
	        $subId_1
	    ) = $h->SUBTOPICS_QUERY(
	        ['root'=>true]
	    )(
	        '0.subtopics',
	        '0.subtopics.0.id'
	    );

	    // safety check to verify if both were really inserted
	    $this->assertCount(2,$subtopics);

		//here we remove the first added subtopic
	    $ret = $h->SUBTOPIC_DELETE(['id'=>$subId_1])();

	    list(
	        $subtopics,
	        $name
	    ) = $h->SUBTOPICS_QUERY(['root'=>true])(
	        '0.subtopics',
	        '0.subtopics.0.name'
	    );

	    // and finally we check to see if only the second subtopic is present
	    // it's worth noticing that there can be more than one root subtopic.
	    // that's why we search with 0 (root) . subtopics (it's children) . 0 (first child) . name
	    $this->assertCount(1,$subtopics);
	    $this->assertEquals('sub2',$name);
	}
```

To run tests without errors, I need to clean the cache. I've not inspected the reasons on that yet, but it feels like a bug on the generation of the cached graphql types. Anyway, I just use a command like this everytime:

```
bin/console cache:clear --env=test;phpunit tests/AppBundle/GraphQL/Subtopics/Mutations/DeleteTest.php
```

I encourage you to open SubtopicsTestHelper and follow and understand the 'proccessResponse' method (Reis\GraphQLTestRunner\Runner\Runner::processGraphQL). There you will be able to see the GraphQL call happening and the json path component being wrapped in the returning funcion.

Some words about Optimization
=============================

	- Optimização
		Optimize human time, not machine time
			Premature optimization is one of the cardinal sins of programming
			I realized that sticking to one query per resolver actually optimized a far more important parameter:
				how many hours I spent writing and rewriting code every time the API changed.
			É mais barato ter menos mão de obra e mais infra.
				Principalmente se temos clareza dos gargalos.

	Depois disso pense em otimizar performance
		Mensure constantemente.
		Melhor do que otimizar queries a bancos é usar cache e batching.
			Há dúvidas se muitas queries simples são realmente melhores que uma grande.
				Mas, independentemente disso, o ganho em ter um código limpo é
					poder reduzir mão de obra, e a
					facilidade e velocidade de manutenção e evolução.
			De qualquer forma não tome isso por verdade. Meça, teste e escolha seu caminho.

	Performance:

		https://github.com/overblog/GraphQLBundle/blob/master/Resources/doc/definitions/debug/index.md

		---------
			overblog_graphql:
			    definitions:
			        show_debug_info: true
		---------
			A toolbar do Symfony funciona melhor

	https://www.youtube.com/watch?v=c35bj1AT3X8 - Excelente, mostra o funcionamento de JS assíncrono e Promisses para então explicar o DataLoader.

	https://www.youtube.com/watch?v=OQTnXNCDywA - Lee Byron explicando o código fonte do DataLoader.js

	Ferramentas de monitoramento:
		http://www.apollodata.com/optics

	Referências
		https://scaphold.io/community/blog/apollo-optics-and-your-graphql-server/
		https://dev-blog.apollodata.com/optimizing-your-graphql-request-waterfalls-7c3f3360b051

	Bela Referência:
		https://dev-blog.apollodata.com/how-to-build-graphql-servers-87587591ded5

	Backend as a Service
		https://scaphold.io/

repository: henriquechaves/palestraGraphQL

repo_name: henriquechaves/palestraGraphQL


text:
Contributing
===========

## Setting up an IDE

The project should import into IntelliJ like normal sbt builds.

## Testing

- `test`: to run fast unit tests
- `scripted`: to run slow integration tests
- `scripted sbt-scalafix/$NAME`: to run an individual integration test.
  Use tab completions to expan `$NAME`.

## Formatting

Be sure to run the `bin/scalafmt` script to ensure code formatting is OK.  Pro
tip, `./bin/scalafmt --diff` formats only the files that have changed from the
main branch.  You can read more about it at http://scalafmt.org

## TL;DR

If you are unsure about anything, don't hesitate to ask in the [Discord channel](https://discord.gg/8AHaqGx3Qj).

repository: scalacenter/sbt-scalafix

gha_language: Scala

repo_name: scalacenter/sbt-scalafix


text:
# gulp-ccr-each-dir

Iterates each sub directories and pass to sub tasks. A cascading configurable gulp recipe for [gulp-chef](https://github.com/gulp-cookery/gulp-chef).

## Install

``` bash
$ npm install --save-dev gulp-chef gulp-ccr-each-dir
```

## Recipe

Stream Array (from [gulp-cheatsheet](https://github.com/osscafe/gulp-cheatsheet) p.2)

## Ingredients

* [merge-stream](https://github.com/grncdr/merge-stream)

* [each](https://github.com/gulp-cookery/gulp-ccr-each)

## Type

[Stream Processor](https://github.com/gulp-cookery/gulp-chef#writing-stream-processor)

## API

### config.src

The directory path to iterate its sub directories. Inject "`config.dir`" and "`config.path`" context to sub tasks.

#### [expose] config.dir

The name of the sub directory.

#### [expose] config.path

The canonical path of the sub directory.

## Usage

``` javascript
var gulp = require('gulp');
var chef = require('gulp-chef');

var meals = chef({
    src: 'src/',
    dest: 'dist/',
    'each-dir': {
        src: 'modules/',
        browserify: {
            bundle: {
                entry: '{{dir}}/index.js',
                file: '{{dir}}.js'
            }
         }
    }
});

gulp.registry(meals);
```

## References

* [Generating a file per folder](https://github.com/gulpjs/gulp/blob/master/docs/recipes/running-task-steps-per-folder.md)

## License
[MIT](https://opensource.org/licenses/MIT)

## Author
[Amobiz](https://github.com/amobiz)

repository: gulp-cookery/gulp-ccr-each-dir

repo_name: gulp-cookery/gulp-ccr-each-dir


text:
PEP: 3122 Title: Delineation of the main module Version: $Revision$
Last-Modified: $Date$ Author: Brett Cannon Status: Rejected Type:
Standards Track Content-Type: text/x-rst Created: 27-Apr-2007
Post-History:

.. attention:: This PEP has been rejected. Guido views running scripts
within a package as an anti-pattern \[\#guido-rejection\]\_.

Abstract
========

Because of how name resolution works for relative imports in a world
where PEP 328 is implemented, the ability to execute modules within a
package ceases being possible. This failing stems from the fact that the
module being executed as the "main" module replaces its `__name__`
attribute with `"__main__"` instead of leaving it as the absolute name
of the module. This breaks import's ability to resolve relative imports
from the main module into absolute names.

In order to resolve this issue, this PEP proposes to change how the main
module is delineated. By leaving the `__name__` attribute in a module
alone and setting `sys.main` to the name of the main module this will
allow at least some instances of executing a module within a package
that uses relative imports.

This PEP does not address the idea of introducing a module-level
function that is automatically executed like PEP 299 proposes.

The Problem
===========

With the introduction of PEP 328, relative imports became dependent on
the `__name__` attribute of the module performing the import. This is
because the use of dots in a relative import are used to strip away
parts of the calling module's name to calculate where in the package
hierarchy an import should fall (prior to PEP 328 relative imports could
fail and would fall back on absolute imports which had a chance of
succeeding).

For instance, consider the import `from .. import spam` made from the
`bacon.ham.beans` module (`bacon.ham.beans` is not a package itself,
i.e., does not define `__path__`). Name resolution of the relative
import takes the caller's name (`bacon.ham.beans`), splits on dots, and
then slices off the last n parts based on the level (which is 2). In
this example both `ham` and `beans` are dropped and `spam` is joined
with what is left (`bacon`). This leads to the proper import of the
module `bacon.spam`.

This reliance on the `__name__` attribute of a module when handling
relative imports becomes an issue when executing a script within a
package. Because the executing script has its name set to `'__main__'`,
import cannot resolve any relative imports, leading to an `ImportError`.

For example, assume we have a package named `bacon` with an
`__init__.py` file containing::

from . import spam

Also create a module named `spam` within the `bacon` package (it can be
an empty file). Now if you try to execute the `bacon` package (either
through `python bacon/__init__.py` or `python -m bacon`) you will get an
`ImportError` about trying to do a relative import from within a
non-package. Obviously the import is valid, but because of the setting
of `__name__` to `'__main__'` import thinks that `bacon/__init__.py` is
not in a package since no dots exist in `__name__`. To see how the
algorithm works in more detail, see `importlib.Import._resolve_name()`
in the sandbox \[\#importlib\]\_.

Currently a work-around is to remove all relative imports in the module
being executed and make them absolute. This is unfortunate, though, as
one should not be required to use a specific type of resource in order
to make a module in a package be able to be executed.

The Solution
============

The solution to the problem is to not change the value of `__name__` in
modules. But there still needs to be a way to let executing code know it
is being executed as a script. This is handled with a new attribute in
the `sys` module named `main`.

When a module is being executed as a script, `sys.main` will be set to
the name of the module. This changes the current idiom of::

if **name** == '**main**': ...

to::

import sys if **name** == sys.main: ...

The newly proposed solution does introduce an added line of boilerplate
which is a module import. But as the solution does not introduce a new
built-in or module attribute (as discussed in `Rejected Ideas`\_) it has
been deemed worth the extra line.

Another issue with the proposed solution (which also applies to all
rejected ideas as well) is that it does not directly solve the problem
of discovering the name of a file. Consider `python bacon/spam.py`. By
the file name alone it is not obvious whether `bacon` is a package. In
order to properly find this out both the current direction must exist on
`sys.path` as well as `bacon/__init__.py` existing.

But this is the simple example. Consider `python ../spam.py`. From the
file name alone it is not at all clear if `spam.py` is in a package or
not. One possible solution is to find out what the absolute name of
`..`, check if a file named `__init__.py` exists, and then look if the
directory is on `sys.path`. If it is not, then continue to walk up the
directory until no more `__init__.py` files are found or the directory
is found on `sys.path`.

This could potentially be an expensive process. If the package depth
happens to be deep then it could require a large amount of disk access
to discover where the package is anchored on `sys.path`, if at all. The
stat calls alone can be expensive if the file system the executed script
is on is something like NFS.

Because of these issues, only when the `-m` command-line argument
(introduced by PEP 338) is used will `__name__` be set. Otherwise the
fallback semantics of setting `__name__` to `"__main__"` will occur.
`sys.main` will still be set to the proper value, regardless of what
`__name__` is set to.

Implementation
==============

When the `-m` option is used, `sys.main` will be set to the argument
passed in. `sys.argv` will be adjusted as it is currently. Then the
equivalent of `__import__(self.main)` will occur. This differs from
current semantics as the `runpy` module fetches the code object for the
file specified by the module name in order to explicitly set `__name__`
and other attributes. This is no longer needed as import can perform its
normal operation in this situation.

If a file name is specified, then `sys.main` will be set to
`"__main__"`. The specified file will then be read and have a code
object created and then be executed with `__name__` set to `"__main__"`.
This mirrors current semantics.

Transition Plan
===============

In order for Python 2.6 to be able to support both the current semantics
and the proposed semantics, `sys.main` will always be set to
`"__main__"`. Otherwise no change will occur for Python 2.6. This
unfortunately means that no benefit from this change will occur in
Python 2.6, but it maximizes compatibility for code that is to work as
much as possible with 2.6 and 3.0.

To help transition to the new idiom, 2to3 \[\#2to3\]\_ will gain a rule
to transform the current `if __name__ == '__main__': ...` idiom to the
new one. This will not help with code that checks `__name__` outside of
the idiom, though.

Rejected Ideas
==============

`__main__` built-in
-------------------

A counter-proposal to introduce a built-in named `__main__`. The value
of the built-in would be the name of the module being executed (just
like the proposed `sys.main`). This would lead to a new idiom of::

if **name** == **main**: ...

A drawback is that the syntactic difference is subtle; the dropping of
quotes around "**main**". Some believe that for existing Python
programmers bugs will be introduced where the quotation marks will be
put on by accident. But one could argue that the bug would be discovered
quickly through testing as it is a very shallow bug.

While the name of built-in could obviously be different (e.g., `main`)
the other drawback is that it introduces a new built-in. With a simple
solution such as `sys.main` being possible without adding another
built-in to Python, this proposal was rejected.

`__main__` module attribute
---------------------------

Another proposal was to add a `__main__` attribute to every module. For
the one that was executing as the main module, the attribute would have
a true value while all other modules had a false value. This has a nice
consequence of simplify the main module idiom to::

if **main**: ...

The drawback was the introduction of a new module attribute. It also
required more integration with the import machinery than the proposed
solution.

Use `__file__` instead of `__name__`
------------------------------------

Any of the proposals could be changed to use the `__file__` attribute on
modules instead of `__name__`, including the current semantics. The
problem with this is that with the proposed solutions there is the issue
of modules having no `__file__` attribute defined or having the same
value as other modules.

The problem that comes up with the current semantics is you still have
to try to resolve the file path to a module name for the import to work.

Special string subclass for `__name__` that overrides `__eq__`
--------------------------------------------------------------

One proposal was to define a subclass of `str` that overrode the
`__eq__` method so that it would compare equal to `"__main__"` as well
as the actual name of the module. In all other respects the subclass
would be the same as `str`.

This was rejected as it seemed like too much of a hack.

References
==========

.. \[\#2to3\] 2to3 tool (http://svn.python.org/view/sandbox/trunk/2to3/)
\[ViewVC\]

.. \[\#importlib\] importlib
(http://svn.python.org/view/sandbox/trunk/import\_in\_py/importlib.py?view=markup)
\[ViewVC\]

.. \[\#guido-rejection\] Python-Dev email: "PEP to change how the main
module is delineated"
(https://mail.python.org/pipermail/python-3000/2007-April/006793.html)

Copyright
=========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

repository: erdal-pb/opep

repo_name: erdal-pb/opep


text:
# Advanced "linking"

In the previous (first) example, I taught you how to make the game call your C (by exposing C functions through EA and then doing the same thing as you'd do with asm). But how do we do the opposite? How do we call the already existing parts of the game with C?

The game was almost definitely written in C (crazy, I know). This doesn't really mean too much to us; but it does mean that each function follow ARM C calling conventions. In other words, we don't need to do anything crazy to interface our code with the game's functions (we just need to let the compiler and/or `lyn`/EA know where those functions are).

In this chapter, I'll use the simple example of the "`GetGameTime`" function (This is just the name I gave it, there's no "official" name to it), located in FE8U at `0x08000D28`, that returns the game time (in frames) and takes no arguments. Its signature is the following:

```c
int GetGameTime(void);
```

_**Note**: when calling game functions from C, unlike from ASM, you want to declare their signature. The compiler needs this information to behave properly._

## Declaring pointers in the source

If you did some ASM Hacking before (you probably did), you may have done the following: you defined the address of a game routine you wanted to call, loaded that and "`bl` to `bx`"ed to it (or "`.short 0xF800`"ed to it). And you totally can do a similar thing in C too! I even kinda did that in the example when writing to event slots. Those were simple int pointers, but you can do the same thing with function pointers, look:

```c
static const int(*GetGameTime)() = (int(*)()) (0x08000D29); // note the thumb bit here.

static int* const sEventSlot = (int* const) (0x030004B8);

/*!
 * Gets game time (in frames) and stores it to event slot C.
 */
void asmc_get_time(void) {
	sEventSlot[0xC] = GetGameTime();
}
```

I want to show you that this indeed works. Let's look at the asm (The comments are my addition, so that things were a bit more clear):

```arm
asmc_get_time:
	push {r4, lr}

	ldr  r3, .L2   @ =0x08000D29
	bl   .L4       @ bx r3

	ldr  r3, .L2+4 @ =0x030004B8+0xC*sizeof(int)
	str  r0, [r3]

	pop  {r4}
	pop  {r0}
	bx   r0
.L3:
	.align 2
.L2:
	.word 134221097 @ 0x08000D29
	.word 50332904  @ 0x030004B8+0xC*sizeof(int)
	.code 16
	.align 1
.L4:
	bx   r3
```

Yup, that's your good old `bl` to `bx` trick.

_**Note**: As it was noted in replies below, you can also achieve this by using raw expressions or `#define`s instead of static constants, but the general concept is the same._

Ok so that works, we're done now. Right? Well sure you can go ahead and do it that way if you want I, won't judge (Ok maybe I will just a teeny tiny bit).

I'll ask you this: What if @7743 wants to port your hack to FE8J (or FE8E lol). Since you basically "hardcoded" the game's addresses in your source you need to edit all of that to support FE8J. This means you end up with two different code bases that you have to update, which is kind of eh (imagine doing this for hack with 5-6 files of a couple hundred lines each).

Of course this can be mitigated by using a header for all the declaration of the game functions there, and using ifdef guards to allow to support for mutiple games, which you can also totally do and I won't judge you (or, again, maybe I will). In my opinion this starts to get messy and I think we can do something more elegant.

## Creating a "library" for interfacing with the game

_**Note**: for the sake of this guide, I will demonstrate making a new library and reference from scratch. For "real" hacking I use and provide [a fairly big existing library and reference](https://github.com/StanHash/FE-CLib) that I recommend you use too after you understand how it works._

So here's the deal: we are going to decouple the *declaration* of what the game provides us (including the *signature* of functions), and the *definition* of where it is.

The *declaration* part, you probably guessed it, we are going to put that in a good old C *header* file. Let's call it, `gbafe.h`. It would contain function declarations, of course, but also maybe eventually structure definitions for data from the game (like say, the [unit struct](https://github.com/StanHash/FE-CLib/blob/master/include/gbafe/unit.h)) and object declarations (such as the active unit pointer). From that all we need to do in our source file is to `#include` it and use it.

Here's the header file containing "`GetGameTime`" and "`gEventSlot`" declarations:

```c
#ifndef GBAFE_H
#define GBAFE_H

extern int gEventSlot[];

int GetGameTime(void);

#endif // GBAFE_H
```

_**Note** about why I went from `sEventSlot` to `gEventSlot`: `s` stands for static and `g` stands for global. It's a relatively common naming convention in the world of C programming (I think, at least the decomp uses it)._

Ok so I guess now I need to explain a bit more about symbols and how C names translate to them. See the `gEventSlot` declaration? It will generate a (undefined) `gEventSlot` symbol. The value of that symbol is expected to be the *address of the object it represents*.

This is why I decided to declare `gEventSlot` as an array and *not* a pointer, because if I declared a pointer, It would expect as value the address of a pointer, but the address I want to give it is the address of the event slot value array (`0x030004B8`), and the data there isn't a pointer, but the value of slot 0 (which so happens to be 0 most of the time, not really a useful pointer).

Here's the C file we'll use:

```c
#include "gbafe.h"

/*!
 * Gets game time (in frames) and stores it to event slot C.
 */
void asmc_get_time(void) {
    gEventSlot[0xC] = GetGameTime();
}
```

Generated ASM:

```arm
asmc_get_time:
	push  {r4, lr}
	bl    GetGameTime
	ldr   r3, .L2
	str   r0, [r3, #48]
	pop   {r4}
	pop   {r0}
	bx    r0
.L2:
	.word gEventSlot
```

Notice that the generated code only uses a standard `bl`, which may cause problems (mainly because the main game code isn't usually within "`bl`-range" of our inserted hacks), so how do we make it a safer `bl` to `bx`?

Note: if you are using `lyn`, it is even more important to transform `bl`s into `bl` to `bx`es, since `lyn` can't *relocate* (that's the term for when a reference in an object file (aka a *relocation*) is being resolved properly) *relative* references (such as a `bl`) to external symbols. This comes from the simple fact the `lyn` doesn't know where the object it links will be (since that's for EA to decide).

So, there's two ways we can do that: either during linking, or during compilation.

#### long calls during linking

`lyn` comes with the `-longcalls` option, This tells `lyn` that when it finds itself in a situation where it can't properly relocate a symbol because the relocation isn't absolute (for example: when relocating a `bl`), it is allowed to add extra code (a "veneer") to "make" it absolute. Example:

- Without `-longcalls`:

```
PUSH
ORG (CURRENTOFFSET+$1); asmc_get_time:
POP
SHORT $B510
SHORT ((((GetGameTime-4-CURRENTOFFSET)>>12)&$7FF)|$F000) ((((GetGameTime-4-CURRENTOFFSET)>>1)&$7FF)|$F800)
SHORT $4B02 $6318 $BC10 $BC01 $4700
POIN gEventSlot
```

- With `-longcalls`:

```
PUSH
ORG (CURRENTOFFSET+$1); asmc_get_time:
POP
{
PUSH
ORG (CURRENTOFFSET+$15); _LP_GetGameTime:
POP
SHORT $B510
SHORT ((((_LP_GetGameTime-4-CURRENTOFFSET)>>12)&$7FF)|$F000) ((((_LP_GetGameTime-4-CURRENTOFFSET)>>1)&$7FF)|$F800)
SHORT $4B02 $6318 $BC10 $BC01 $4700
POIN gEventSlot
SHORT $4778 $46C0
WORD $E59FC000 $E12FFF1C
POIN GetGameTime
}
```

Whoah that's a lot of new stuff. In realtity that's only 16 extra bytes (2 SHORTS, 2 WORDS, 4 BYTES at the very end of the thing). But there's also a new symbol, `_LP_GetGameTime` ~~that probably shouldn't be here because it should know how to optimize it away so that's a bug for me to fix but in this particular case it's useful for getting my point accross~~, that replaces the regular `GetGameTime`, and this time it is properly defined and it even doesn't pollute the global EA namespace since it's enclosed in curly brackets.

The extra 16 bytes at the end are a piece of asm that "transforms" the `bl` into a `bl` to `bx`. It's different to a regular `bl` to `bx` because the setup (loading of the target address) here is done *after* the `bl`.

Anyway, since the call is transformed, `lyn` could relocate the `GetGameTime` symbol (if it knew where it was, but we'll get to that). Even better: we aren't limited by the potential limitation of the range of the `bl` instruction (+-4MB). Mission accomplished... But I think we can be better.

#### long calls during compilation

There's multiple ways to go on this:

- We could use the compiler and target-specific function attributes in our header to specifiy we want to long call:

  ```c
  int GetGameTime(void) __attribute__ ((long_call));
  ```

  That's honestly only really okay for quick testing: if we start adding other signatures it can get tedious really fast (not to mention pretty ugly).

- We could use the compiler and target-specific `#pragma long_calls`:

  ```c
  #pragma long_calls

  int GetGameTime(void);

  // as many other function declaration as you want

  #pragma long_calls_off
  ```

  This `long_calls` pragma applies the `long_call` attribute automatically to all function declarations encountered until the next `long_calls_off` pragma. This is fine and I've been using this method for the last year now (this what was originally recommended in this guide too), but I feel like the next and last one may actually be the slightly better way.

- We could use the compiler and target-specific program option `-mlong-calls`, which makes *all* references to outside code a "long call". This may be a bit unnecessary for function calls accross our own files (that would probably be inserted close together aka within `bl`-range), but this allows us to move the details related to the limitations of our setup *outside* of the source itself which I feel may be preferable.

  We can just add `-mlong-calls` to the definition of the `CFLAGS` variable in our makefile.

In all three cases we get this ASM:

```arm
asmc_get_time:
	push  {r4, lr}

	ldr   r3, .L2
	bl    .L4

	ldr   r3, .L2+4
	str   r0, [r3, #48]

	pop   {r4}
	pop   {r0}
	bx    r0

.L2:
	.word GetGameTime
	.word gEventSlot

.L4:
	bx    r3
```

A good old `bl` to `bx`. Wonderful. And it does look better than what `lyn` did, which makes sense since we tackled the problem at an earlier point in the compilation process.

Ok so now comes the *definition* part. I'll show you *two* ways to handling that: through a "reference object" (my recommendation), and through EA (for the lazy ones).

### Definition through a "reference object"

In this method, we *define* the symbols we need in an external object file: the "reference object". This file and *only* this file (in theory) is game-specific, since this is the only file where we will have actuall addresses in it.

This source file will *not* be a C file, but a good old ASM file. We won't be putting any actual ASM in it though, we will rather be abusing assembler directives to *generate an object file containing the symbol definitions we need*.

Here's mine:

```arm
.macro SET_FUNC name, value
	.global \name
	.type   \name, %function
	.set    \name, \value
.endm

.macro SET_DATA name, value
	.global \name
	.type   \name, %object
	.set    \name, \value
.endm

SET_DATA gEventSlot,  (0x030004B8)

SET_FUNC GetGameTime, (0x08000D28+1)
```

Note that I'm defining through helper macros I also defined. For now it seems kinda redundant but later when we start to add dozens of extra functions and/or references to data, you'll thank me.

Anyway, I called it `fe8u.s` (I'm willingly choosing a different extension to the one for generated asm from C files, as it allows me to differentiate between generated asm and hand-written asm (gets useful when writing `clean` rules in the makefile later)).

Hey! We don't even have a rule for making object files from asm in our makefile (this probably means that I'm doing a good job), so let's remedy this. Add this to your makefile:

```makefile
# ASM to OBJ
%.o: %.s
	$(AS) $(ARCH) $< -o $@
```

Now we can `make fe8u.o` (we can also `make fe8u.lyn.event`, but this will just generate a blank file, as there isn't any data nor local symbols defined in it, and `lyn` doesn't expose absolute symbols).

Ok so now what? We have a nice object file with our symbols defined in it, and a nice header with our symbols declared... but how do we use all that?

The awnser is: we need to link the object file generated from that C file with `fe8u.o`. When we do that, the symbols referenced by the C object will be sustitued by their definition from the `fe8u.o` object. To do that, we simply pass them both to `lyn`:

```sh
$ ./lyn <myfile>.o fe8u.o
```

It gives me this:

```
PUSH
ORG (CURRENTOFFSET+$1); asmc_get_time:
POP
SHORT $B510 $4B04 $F000 $F80A $4B03 $6318 $BC10 $BC01 $4700 $46C0
BYTE $29 $0D $00 $08 $B8 $04 $00 $03
SHORT $4718 $46C0
```

Sweet. We linked our objects together.

#### (Makefile) Automatically making & linking the reference object

I'm going to assume we need that reference object for *all* of our `lyn` links. That way we can take the rule for making `.lyn.event` files and modify it accordingly:

```makefile
# OBJ to EVENT rule
%.lyn.event: %.o fe8u.o
	$(LYN) $< fe8u.o > $@
```

If you aren't familiar with makefiles: I added a second *dependency* to the pattern rule for generating `.lyn.event` from `.o` files. This means that each time you require a `.lyn.event`, it will first make sure that both the corresponing `.o` file and the `fe8u.o` file are *up to date* (aka younger than their own dependencies).

This story about makefiles & up-to-dateness will come again later when we will learn how to generate dependency definitions for our C files (that are including other files, on which the source *depends* on).

Anyway, this simple change makes sure that the reference is built before building the `.lyn.event` file.

Also the `lyn` call was changed to include the reference too.

You could also define a Make variable that holds the reference object to link against. For example:

```makefile
# reference object
LYNREF := fe8u.o
```

And the rule then becomes

```makefile
# OBJ to EVENT rule
%.lyn.event: %.o $(LYNREF)
	$(LYN) $< $(LYNREF) > $@
```

### Definition through EA

When `lyn` can't relocated something, it doesn't fail. Instead it just writes the name of the targetted symbol in place, letting EA handle it and probably fail in its place.

But we can also take this to our advantage. If we get our object and `lyn` it, we get this:

```
PUSH
ORG (CURRENTOFFSET+$1); asmc_get_time:
POP
SHORT $B510 $4B04 $F000 $F80A $4B03 $6318 $BC10 $BC01 $4700 $46C0
POIN GetGameTime gEventSlot
SHORT $4718 $46C0
```

Direct reference to the missing symbols can be seen. So what do we do? Well, we `#define` them!

```
#define GetGameTime "(0x08000D28+1)"
#define gEventSlot  "(0x030004B8)"
```

ez. Know that since EA 11.1 (the version in which I did stuff shh that's a secret) (Note: this is *not* yet part of ColorzCore), A feature exists that allows us to do the following:

```
GetGameTime = 0x08000D28+1
gEventSlot  = 0x030004B8
```

Anyway, done. Put that anywhere (if you used `#define`, it needs to be brefore the `lyn` output), and the `lyn` output *will* assemble and link correctly. Hurray!

This is obviously easier than to go through all the hassle of making another object file and linking it via a `lyn` call and all, and also way more clean that the brute way of defining const function pointers. I'm guessing most people reading this ~~(that's probably not a lot of people)~~ will choose to use this method over the other two.

The issue I find with this method is in the hack distribution side of things.

But still, I think the less EA does, and the more processing you can have done before the "EA" phase, the better it is (EA does already enough as is).

# Integrating ASM & C

Ok, so you already know how to call C from Events (`ASMC`, see our first example) and probably have an idea of how to call C with EA `Extensions/Hack Installation.txt` Macros (`jumpToHack`, `callHack_rx`, ...), since it's basically the same in a different context.

Here we are going to see how to integrate your C code & regular "raw" ASM. This will be probably most useful when setting up more elaborate hooks and you need to setup some very precise ASM injections that you just *can't* have the C compiler generate elegantly.

## Calling C from ASM

The solution is pretty simple really, just write some ASM, and somewhere in it you `bl` or `bl` to `bx` to the C function, as if its name was a simple label in your ASM:

```arm
.thumb

some_asm:
	push {r0-r3}

	@ let's say we are setting up arguments for our function
	mov r0, r2
	mov r1, r3

	ldr r3, =some_c_function
	bl BXR3

	pop {r0-r3}

	@ let's assume this is the hooked-into routine's old code that we replaced
	nop
	nop
	nop

	ldr r3, =#0x800DEAD @ return location
BXR3:
	bx r3
```

*Then*, instead of throwing that into "Assemble ARM.bat", just `make` an object file, and then we get it through `lyn`. We get this:

```
SHORT $B40F $1C10 $1C19 $4B04 $F000 $F805 $BC0F $46C0 $46C0 $46C0 $4B01 $4718
POIN some_c_function
BYTE $AD $DE $00 $08
```

Hey, `some_c_function` got referenced in EA! This works, we can now get our C function through `lyn` too and we have something like this:

```
PUSH
ORG (CURRENTOFFSET+$1); some_c_function:
POP
SHORT $Whatever $46C0
```

If we put those together we get something that assembles correctly and everything is nice yay.

## Calling ASM from C

So in this part, we are going back to messing with ASM a bit more. The C compiler already generates pretty "open" (as in: easily interfacable) object files, but this isn't necessarily the case for ASM.

For C or, to be more specific, the linker to know how to call your asm, you need to *expose* the symbol pointing to the part of your asm you want to call (and also it needs the symbol to be of "function" type, so that it knows you're not accidentally calling data as a routine)

So here's the two *assembler directives* we are going to use:

```arm
.global <name>
.type   <name>, %function
```

If you paid attention earlier (when I told you about making object files containing information about the game), you'll notice that those are the same directives I used then.

`.global <name>` *exposes* the symbol to the outside (in other words, external object files can "see" it).

`.type <name>, %function` marks the symbol as being of "function" type, so that the outside knows that this symbol can be the target of a function call.

Here's some example ASM:

```arm
.thumb

.global get_magic_number
.type   get_magic_number, %function

get_magic_number:
	mov r0, #42
	bx  lr
```

Now, our C file:

```c
#include "gbafe.h"

int get_magic_number(void);

void asmc_get_magic_number(void) {
	gEventSlot[0xC] = get_magic_number();
}
```

Assemble/compile and lyn both, `#include` them and it should work!

## Calling conventions

[I wrote another fairly big document on this subject](https://feuniverse.us/t/doc-asm-conventions/3350?u=stanh), so I'll just go over the important details here.

C function signatures hold the information neccessary for the C compiler to know how to handle the called function's parameters & return value. Simple parameters & return values are pretty simple to understand: each parameter corresponds to a register (up to `r3`), and the return value to `r0` after the function call.

There are some ambiguous cases though, and we can tell they're ambiguous since beginner ASM hackers have different ways of tackling them.

The first (obvious) one is the case where you have more than 4 parameters. Basic logic would dictate you continue onto `r4`, `r5` and so on... But then what happens when you reach `r11`? You can't use `r12` since it can be used to redirect the call (see the aforementionned doc for details), and after that its the stack pointer...

The stack! Yes that's exactly where extra arguments are going to be. In fact, convention doesn't wait until `r11` for that, after `r3` (fourth parameter), the arguments are stored at `[sp+00]`, `[sp+04]`, etc.

So a function with the following signature

```c
void some_function(int a, int b, int c, int x, int y);
```

Will have the following parameter map:

- `r0` = `int a`
- `r1` = `int b`
- `r2` = `int c`
- `r3` = `int x`
- `[sp+00]` = `int y`

Second case: we have a parameter that can't fit in a register (or any word in general). That one is kinda simple: we split the parameter in mutiple parts, and each part is its own parameter. It is the called function's responsability to interpret the different parts correctly.

For example:

```c
struct xyz { short x, y, z; }
void some_function(struct xyz positionInThreeDee, int b);
```

Will have the following parameter map:

- `r0` = `positionInThreeDee` part 1 (`xyz.x` & `xyz.y`)
- `r1` = `positionInThreeDee` part 2 (`xyz.z` & padding)
- `r2` = `int b`

Third case: we have a return value that can't fit in a register *and isn't either a `long long`, a `double`, a `long double`, or a complex* (in those cases, the return value is split using the same rules as parameters, and can use registers `r0` through `r3`).

In this case, an extra parameter is prepended (added before all the others) to the list: it's a pointer holding the address of where to write the return value. The function also still returns that address.

For example:

```c
struct BigStruct some_function(int a);
```

Will have the following parameter map:

- `r0` = `struct BigStruct*` (somewhere to write the return value)
- `r1` = `int a`

There's probably other corner case scenarios I forgot to mention here. If you have any doubt: I recommend you write a test function, compile it to asm and analyse the result for yourself. ~~Or read the doc~~.

license: Unlicense

repository: StanHash/CHax-Guide-Files

repo_name: StanHash/CHax-Guide-Files


text:
# Demo Project for "İstanbul" issue

This is a demonstration to showcase the slugification problem of "İstanbul".

As originally discussed in [a thread in Wagtail's support channel](https://wagtailcms.slack.com/archives/C81FGJR2S/p1570706995063800).

## Setup

This repo contains 2 projects, both using official procedures:

1. was created using wagtail's _Getting Started_ (`mysite_wagtail`)
2. was created using django's `django-admin.py startproject` and `./manage.py startapp` (`mysite_django`).

```sh
$ pipenv install
# Cd into either mysite_django or mysite_wagtail
$ cd mysite_django
$ ./manage.py test
```

In both projects, the tests fail. Why they do and what the expected behaviour would
be, is explained in detail below.

## The issue

When wanting to create a new page with the title `Hello İstanbul`, the slugification
with `allow_unicode=True` fails. Further analysis is below the _Steps to reproduce_.

## Wagtail: Steps to reproduce in Webinterface

```sh
$ ./manage.py migrate
# Create a user to access the admin interface.
$ ./manage.py createsuperuser
# And off we go!
$ ./manage.py runserver
```

1. Log into the admin with your superuser: [http://localhost:8000/admin/](http://localhost:8000/admin/).
2. Open up Home in the explorer
3. Click _Add Child Page_
4. Set `Hello İstanbul` as title
5. Press _Save Draft_
6. The error message "The page could not be created due to validation errors" appears
   and the `Promote` tab has a badge with a 1 in it.
7. Click on the `Promote` tab
8. You will see an error message: `Enter a valid 'slug' consisting of Unicode letters, numbers, underscores, or hyphens.`.

## Expected behaviour

The lowercased version of the uppercase `İ` (latin capital letter i with dot above)
should work as a slug.

## Analysis

First we analyze the letter, then we try to reason why this fails with wagtail/django.

### Letter

What kind of character is this?

```python
In [1]: import unicodedata
In [2]: unicodedata.name('İ')
Out[2]: 'LATIN CAPITAL LETTER I WITH DOT ABOVE'
```

Also, we can see that this letter only consists of a single unicode character.

```python
In [3]: [unicodedata.name(character) for character in 'İ']
Out[3]: ['LATIN CAPITAL LETTER I WITH DOT ABOVE']
```

Now, we let Python create the lowercase version

```python
In [5]: li = 'İ'.lower()
In [6]: li
Out[6]: 'i̇'
```

Looks good. Let's see what it is called.

```python
In [7]: unicodedata.name(li)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-70daade6eaaf> in <module>
----> 1 unicodedata.name(li)

TypeError: name() argument 1 must be a unicode character, not str
```

Uh oh. Wait,.. what? So. That means this string consists of two unicode characters.

```python
In [8]: [unicodedata.name(character) for character in li]
Out[8]: ['LATIN SMALL LETTER I', 'COMBINING DOT ABOVE']
```

Indeed it does! So it is a "regular" small "i" with the combining diacritic "dot above".

### Django/Wagtail

So what is going on in Django? As far as I can tell, Wagtail does not have anything
to do with this, as [it simply calls django's slugify()](https://github.com/wagtail/wagtail/blob/e2607f917cc06bdf901ff7a52483ec89ec1115ba/wagtail/core/models.py#L426).
Furthermore it is possible to re-create the exact same error when using only Django,
as is shown in `mysite_django`.

1. Wagtail calls `slugify("Hello İstanbul")`
2. Django slugifies the title
3. Django's SlugField validates the slug
4. If validation succeeds, Django commits it to the database

#### 1. wagtail calls slugify()

We are going to skip this. Looking [very unsuspicious](https://github.com/wagtail/wagtail/blob/e2607f917cc06bdf901ff7a52483ec89ec1115ba/wagtail/core/models.py#L426).

In `mysite_django`, we clone this behaviour in the `mysite_django/demo/models.py`.

#### 2. django slugifies the title

We start by taking a look at [django's slugify](https://github.com/django/django/blob/master/django/utils/text.py#L400).

```python
def slugify(value, allow_unicode=False):
    """
    Convert to ASCII if 'allow_unicode' is False. Convert spaces to hyphens.
    Remove characters that aren't alphanumerics, underscores, or hyphens.
    Convert to lowercase. Also strip leading and trailing whitespace.
    """
    value = str(value)
    if allow_unicode:
        value = unicodedata.normalize('NFKC', value)
    else:
        value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii')
    value = re.sub(r'[^\w\s-]', '', value).strip().lower()
    return re.sub(r'[-\s]+', '-', value)
```

First our string (`"Hello İstanbul"`) is normalized. So it turns each unicode
character into a defined "normalized" form
([python's docs on this explain it well](https://docs.python.org/3.7/library/unicodedata.html#unicodedata.normalize)).

We do `allow_unicode`, so Django uses the normalized C form:

```python
# https://github.com/django/django/blob/master/django/utils/text.py#L400
# value is always "Hello İstanbul"
if allow_unicode:
    value = unicodedata.normalize('NFKC', value)
```

Doing this, does not change anything for us. As it should be, if I understand
Python's docs on this correctly.

```python
In [20]: unicodedata.normalize('NFKC', "Hello İstanbul")
Out[20]: 'Hello İstanbul'
```

Now, the next step in django's slugification is this line:

```python
# https://github.com/django/django/blob/master/django/utils/text.py#L404
value = re.sub(r'[^\w\s-]', '', value).strip().lower()
```

Splitting it up into three steps:

1. Remove anything that is not a space, a (unicode) character or a hypen
2. Strip leading and trailing whitespace
3. Lowercase the entire string

Again, our `value` would be `Hello İstanbul`.  All right!

Then, Django calls `strip`, to free the string of extraneous whitespace:

```python
In [24]: re.sub(r'[^\w\s-]', '', "Hello İstanbul").strip()
Out[24]: 'Hello İstanbul'
```

As expected, everything stays the same.

Now the last step in this very line is to lowercase the string:

```python
In [26]: re.sub(r'[^\w\s-]', '', "Hello İstanbul").strip().lower()
Out[26]: 'hello i̇stanbul'
```

After that, the method replaces all spaces with a dash:

```python
In [34]: re.sub(r'[-\s]+', '-', 'hello i̇stanbul')
Out[34]: 'hello-i̇stanbul'
```

And that's that.

#### 3. Django's SlugField validates the value

This is where things "fail". The SlugField is given the result of the slugification:
`'hello-i̇stanbul'`.

Now the SlugField does [it's validation](https://github.com/django/django/blob/a28d1b38e55cf588cfaae97de6a575d5c9f90a96/django/forms/fields.py#L1178).

If we take a look at the SlugField validator being used (with `allow_unicode = True`), we end up with
the [`slug_unicode_re` regular expression, as defined in `django/core/validators.py`](https://github.com/django/django/blob/master/django/core/validators.py#L244).

```python
# allow for unicode words (\w), dashes (-). \Z is for marking the end of the string.
slug_unicode_re = _lazy_re_compile(r'^[-\w]+\Z')
```

From here, things go awry. That regular expression does not match the output
generated by `slugify("Hello İstanbul")`.

```python
In [30]: type(slug_unicode_re.match('hello-i̇stanbul'))
Out[30]: NoneType
```

Why does it fail? Let's repeat the small analysis we did before and remember that the
lowercase version of `İ` consists of two characters:

```python
# Just a quick reminder:
In [10]: [unicodedata.name(character) for character in unicodedata.normalize('NFKC', 'İ'.lower())]
Out[10]: ['LATIN SMALL LETTER I', 'COMBINING DOT ABOVE']

# Now the gory details:
In [36]: [(character, unicodedata.name(character), slug_unicode_re.match(character)) for character in 'hello-i̇stanbul']
Out[36]:
[('h', 'LATIN SMALL LETTER H', <re.Match object; span=(0, 1), match='h'>),
 ('e', 'LATIN SMALL LETTER E', <re.Match object; span=(0, 1), match='e'>),
 ('l', 'LATIN SMALL LETTER L', <re.Match object; span=(0, 1), match='l'>),
 ('l', 'LATIN SMALL LETTER L', <re.Match object; span=(0, 1), match='l'>),
 ('o', 'LATIN SMALL LETTER O', <re.Match object; span=(0, 1), match='o'>),
 ('-', 'HYPHEN-MINUS', <re.Match object; span=(0, 1), match='-'>),
 ('i', 'LATIN SMALL LETTER I', <re.Match object; span=(0, 1), match='i'>),

 # EUREKA!!
  Which is 100% correct. It should not!
However, that is exactly why the slugification process "fails" in this scenario,
because `slugify()` can be made to produce output that does not validate against
`slug_unicode_re`.

## Summary & Proposed Solution

Summing up: the culprit is the order in which `slugify()` executes the lowercasing.
It does so, _after_ cleaning unwanted characters away. However, as demonstrated,
lowercasing can trigger the creation of unwanted characters. "Unwanted" meaning a
character that can not be validated by the `SlugField`.

There are two ways to solve this:

1. Extend the `slug_unicode_re` to allow for combining diacritics. Or…
2. …change when the lowercasing happens.

Personally I am leaning towards option `2`.

So [this line](https://github.com/django/django/blob/master/django/utils/text.py#L404)
in `django/utils/text.py`'s `slugify`:

```python
value = re.sub(r'[^\w\s-]', '', value).strip().lower()
```

becomes

```python
value = re.sub(r'[^\w\s-]', '', value.lower()).strip()
```

Repeating the proposed solution with our demonstration:

```python
In [37]: re.sub(r'[^\w\s-]', '', 'Hello İstanbul'.lower()).strip()
# Diacritic is now removed, hence we have the lower case latin i
Out[37]: 'hello istanbul'

In [38]: re.sub(r'[-\s]+', '-', 'hello istanbul')
Out[38]: 'hello-istanbul'

In [39]: slug_unicode_re.match('hello-istanbul')
Out[39]: <re.Match object; span=(0, 14), match='hello-istanbul'>
```

### An even deeper dive in (Is Python doing this right?)

Thanks to [Matt Westcott's suggestion](https://wagtailcms.slack.com/archives/C81FGJR2S/p1571400630283400?thread_ts=1570706995.063800&cid=C81FGJR2S)
that it could also be a bug in Python itself, I took a quick dive into the unicode
plane myself.

-   The `İ` is the _Latin Capital Letter I with Dot Above_. It's codepoint is `U+0130`.
    According to the [chart for the Latin Extended-A set](https://www.unicode.org/charts/PDF/U0100.pdf),
    it's lowercase version is `U+0069`.
-   `U+0069` lives in the [C0 Controls and Basic Latin set](https://www.unicode.org/charts/PDF/U0000.pdf).
    Lo and behold: it is the _Latin small letter I_. So a latin lowercase `i`.

Does this really mean that Python is doing something weird here by adding the
_Combining dot above_?

```python
In [8]: [unicodedata.name(character) for character in 'İ'.lower()]
Out[8]: ['LATIN SMALL LETTER I', 'COMBINING DOT ABOVE']
```

I'm in no way a unicode pro, so my assumption might be very naive and wrong.

repository: originell/django-wagtail-turkish-i

gha_language: Python

repo_name: originell/django-wagtail-turkish-i


text:
# update images of the reverse proxy and lets encrypt

Pull the newest image. Stop and delete the old container
create the new container
````bash
#!/bin/bash
docker pull jwilder/nginx-proxy
docker stop nginx-proxy
docker rm nginx-proxy
docker pull jrcs/letsencrypt-nginx-proxy-companion
docker stop letsencrypt
docker rm letsencrypt
$(dirname $0)/createNginxReverseProxyContainer.sh
````

repository: vcpsh/vServer

repo_name: vcpsh/vServer


text:
LiteCoinX
=========
New X11 Coin for breaking LTC.

#### Spec
    Algorithm: X11
    Total Money Supply: 84 Million LTCXs
    Block Time:  45 seconds
    Block Reward: 15 LTCX
    Difficulty Retarget:  DigiShield
    Coin Maturity: 150 blocks
    Confirms: 4
    Premine: 0.05%

#### ANN
  https://bitcointalk.org/index.php?topic=624378

#### Website
  http://www.litecoinx.co

repository: DannyKane/LiteCoinX

repo_name: DannyKane/LiteCoinX


text:
# NLP Patenter
Patent application analysis repository for analyzing chances for success and improving the application. Created by Pierre Alkubeh of Leviasec Corporation. The project is open-source, but we can analyze any patent application for you at www.leviasec.com .

# patent_success_analysis.py
This service checks for:

1. Proper formatting of your patent.

2. Prior art search and returns links with most similar patents

3. Novelty analysis and returns a probability of your patent being novel and therefore accepted by the patent office.

How it works:

The algorithm first extracts the text from the PDF file or submitted text into a string. The string is then analyzed for presence of necessary sections such as a detailed description and for formatting requirements such as the word count for the abstract. A natural-language processing algorithm then extracts  keywords from the application using tokenization and runs a search through the patent office for most similar patents. A final algorithm calculates the similarities between your patent application and the most similar patents to create a novelty score for your patent.

# COMING SOON: STYLE ANALYSIS & WRITING

Patents need to be written in legalese for best protection in court. This algorithm will analyse the writing style in your patent application and rewrite unsatisfactory parts into iron-clad legalese speak.

# COMING SOON: CLAIM ANALYSIS

The claims made are the heart of any patent application. The claim analysis service will analyze the argumentation behind your claim analysis to ensure your success in patenting and defending your invention.

repository: pierre-alkubeh/nlp-patenter

repo_name: pierre-alkubeh/nlp-patenter


text:
# Digital Rain Animation for TFT_eSPI using ESP32, ESP8266

 A library that represents Digital Rain Animation on color displays that support TFT_eSPI
 Search for this library in the Arduino Library Manager and download it or clone it yourself from Github.
 This library is built on TFT_eSPI. Currently only works with ESP32 and ESP3266.

<img width="800" src="https://github.com/0015/TP_Arduino_DigitalRain_Anim/blob/main/misc/image.jpg">

# Updates

 - v1.2.1
    - Added color change features. (BG color, Text Color, Header Char Color)

 - v1.1.1
    - Added the example, DEMO_Generating_Random_Key_Non_FreeRTOS

 - v1.1.0
    - Added the function to generate a random key
    - Added switching to red color for one minute if it has a random key
    - Added the example, DEMO_Generating_Random_Key

# License

 This software is written by Eric Nam and is licensed under The MIT License. Check License.txt for more information.