text:
# [Zabbix PostgreSQL version 11 Native Partitioning](#zabbix-postgresql-native-partitioning)

Table of Contents
=================

   * [Zabbix PostgreSQL version 11 Native Partitioning](#zabbix-postgresql-version-11-native-partitioning)
   * [Table of Contents](#table-of-contents)
      * [Requirements](#requirements)
      * [Zabbix History* and Trends* Tables](#zabbix-history-and-trends-tables)
         * [history & trends](#history--trends)
      * [Zabbix Partitioning Considerations](#zabbix-partitioning-considerations)
      * [Time Series Partitioning](#time-series-partitioning)
         * [PostgreSQL Default Partition](#postgresql-default-partition)
         * [Optional - BRIN versus Btree INDEXES](#optional---brin-versus-btree-indexes)
      * [Prepare PostgreSQL Database](#prepare-postgresql-database)
         * [Install PostgreSQL v11](#install-postgresql-v11)
         * [Prepare Zabbix Database](#prepare-zabbix-database)
         * [Create Empty history* and trends* Tables](#create-empty-history-and-trends-tables)
      * [PostgreSQL Partition Manager Extension (pg_partman)](#postgresql-partition-manager-extension-pg_partman)
         * [Installing pg_partman](#installing-pg_partman)
         * [postgresql.conf](#postgresqlconf)
         * [Create Partitioned Tables](#create-partitioned-tables)
         * [Deleting Designated Partitions](#deleting-designated-partitions)
         * [Partition Maintenance: Creating Future Partitions](#partition-maintenance-creating-future-partitions)
         * [Partition Maintenance: Dropping/expiring old partitions](#partition-maintenance-droppingexpiring-old-partitions)
      * [Set up Zabbix PostgreSQL Partitioning on a Fresh Install](#set-up-zabbix-postgresql-partitioning-on-a-fresh-install)
      * [Troubleshooting](#troubleshooting)
         * [zabbix database user does not have appropriate permissions](#zabbix-database-user-does-not-have-appropriate-permissions)
         * [importing the history table takes forever](#importing-the-history-table-takes-forever)
      * [Change Zabbix history tables from monthly to daily with pgpartman](#change-zabbix-history-tables-from-monthly-to-daily-with-pgpartman)
      * [Zabbix Remote Data Dump](#zabbix-remote-data-dump)
         * [pgdump/pgrestore Manual Mechanism](#pgdumppgrestore-manual-mechanism)
      * [Upgrade Zabbix v3.4 to v4.2 while moving from PostgreSQL v9.6 to PostgreSQL v11](#upgrade-zabbix-v34-to-v42-while-moving-from-postgresql-v96-to-postgresql-v11)
         * [Migrate the Configuration Data](#migrate-the-configuration-data)
         * [Migrate the old history* and trends* data](#migrate-the-old-history-and-trends-data)
      * [Performance Testing](#performance-testing)
         * [pgbench](#pgbench)
         * [EXPLAIN ANALYZE](#explain-analyze)
         * [Zabbix Databases Performance Results](#zabbix-databases-performance-results)
      * [Partitioning Advantages, Mistakes and Problems](#partitioning-advantages-mistakes-and-problems)
         * [Partitioning Advantages](#partitioning-advantages)
         * [Common Partitioning Mistakes](#common-partitioning-mistakes)
      * [Ansible Role](#ansible-role)
      * [References](#references)

---

## [Requirements](#requirements)

* Debian 8 Jessie or Debian 9 Stretch OS Distro
* PostgreSQL version 11
* Zabbix version 3.4 or 4.0 (tested successfully on both)

**NOTE** - This guide was used to _migrate_ a working non-paritioned Zabbix instance to a new instance with partitioned tables ready to ingest the data into the proper partitions. A section was added to address how to set this up for a [Set up Zabbix postgreSQL partitioning on a fresh install](#set-up-zabbix-postgresql-partitioning-on-a-fresh-install)

---

## [Zabbix History* and Trends* Tables](#zabbix-history-and-trends-tables)

* history - `history`, `history_uint`, `history_str`, `history_text`, `history_log`
* trends - `trends`, `trends_uint`

### [history & trends](#history--trends)

* `history` - table that stores all numeric (float) values
* `history_uint` - table that stores all integer values
* `history_log` - table that stores all log values
* `history_text` - table that store all text values
* `history_str` - table that stores all string values
* `trends` - table that stores all numeric (float) values
* `trends_uint` - table that stores all numeric (unsigned integers)

History and trends are the two ways of storing collected data in Zabbix.

Whereas history keeps each collected value, trends keep averaged information on hourly basis and therefore are less resource-hungry.

The general strong advice is to keep history for the smallest possible number of days and that way not to overload the database with lots of historical values.

Instead of keeping a long history, you can keep longer data of trends. For example, you could keep history for 14 days and trends for 5 years.

While keeping shorter history, you will still be able to review older data in graphs, as graphs will use trend values for displaying older data.

Trends is a built-in historical data reduction mechanism which stores minimum, maximum, average and the total number of values per every hour for numeric data types.

Trends usually can be kept for much longer than history. Any older data will be removed by the housekeeper.

When server flushes trend cache and there are already trends in the database for this hour (for example, server has been restarted mid-hour), server needs to use update statements instead of simple inserts. Therefore on a bigger installation if restart is needed it is desirable to stop server in the end of one hour and start in the beginning of the next hour to avoid trend data overlap.

Zabbix updates trends immediately after receipt of new value. Therefore, all information stored in trends is always valid and up-to-date (updated in realtime).

Zabbix generates all graphs from detailed history if period is less than 24 hours, and the trends are used for graphs having period longer than 24 hours.

---

## [Zabbix Partitioning Considerations](#zabbix-partitioning-considerations)

Before performing partitioning in Zabbix, several aspects must be considered:

1. Time partitioning will be used for table partitioning.
2. Housekeeper will not be needed for some data types anymore. This Zabbix functionality for clearing old history and trend data from the database can be controlled in **Administration | General Housekeeper**.
3. The values of History storage period (in days) and Trend storage period (in days) fields in item configuration will not be used anymore as old data will be cleared by the range i.e. the whole partition. They can (and should be) overridden in **Administration | General Housekeeper** - the period should match the period for which we are expecting to keep the partitions which is **monthly**
4. Even with the housekeeping for items disabled, Zabbix server and web interface will keep writing housekeeping information for future use into the housekeeper table. To avoid this, you can add trigger for this table **after you add the data there** (ensure you connect as the `zabbix` user to the `zabbix` database via the following command `$ sudo -u zabbix psql zabbix`):

```SQL
CREATE TRIGGER housekeeper_blackhole
    BEFORE INSERT ON housekeeper
    FOR EACH ROW
    EXECUTE PROCEDURE housekeeper_blackhole();
```

With the following procedure:

```SQL
CREATE OR REPLACE FUNCTION housekeeper_blackhole()
    RETURNS trigger AS
$func$
BEGIN
    RETURN NULL;
END
$func$ LANGUAGE plpgsql;
```

---

## [Time Series Partitioning](#time-series-partitioning)

Partitioning syntax was introduced in PostgreSQL 10. It is very effective for INSERTs and large/slow `SELECT` queries, which makes it suitable for time series logging.

### [PostgreSQL Default Partition](#postgresql-default-partition)

With PostgreSQL version 11 it is possible to create a "default" partition. This stores rows that do not fall into any existing partition's range. This is ideal since the partitioned range might not include specific data which the default will then pick up. This is automatically done with `pg_partman`. From there one can delete all data from the table via the following example:

```SQL
DELETE FROM public.history_p2018_11
```

### [Optional - BRIN versus Btree INDEXES](#optional--brin-versus-btree-indexes)

With PostgreSQL 9.5 a new type of index, [BRIN (Block Range INdex)](https://www.postgresql.org/docs/9.5/brin-intro.html) was introduced. These indexes work best when the data on disk is sorted. Brin only stores min/max values for a range of blocks on disk, which allows them to be small, but which raises the cost for any lookup against the index. Each lookup that hits the index must read a range of pages from disk, so each hit becomes more expensive.

Huge tables benefit from the BRIN index. Adding a BRIN index is fast and very easy and works well for the use case of time series data logging, though less well under intensive update. An INSERTs into BRIN indexes are specifically designed to **not** slow down as the table get bigger, so they perform much better than btree indexes.

In PostgreSQL v11, partitioning offers automatic index creation. You simply create an index on the parent table, and Postgres will automatically create indexes on all child tables. This thus makes partition maintenance much easier!

---

## [Prepare PostgreSQL Database](#prepare-postgresql-database)

1. Download and install the [PostgreSQL Core Distribution](https://www.postgresql.org/download/) that supports Native Partitioning. As of this writing it is PostgreSQL v11.1.
2. Tune `postgresql.conf` to ensure `enable_partition_pruning = on`. The default should be `on`. This enables or disables the query planner's ability to eliminate a partitioned table's partitions from query plans, thus improving performance.
3. Copy the `pg_hba.conf` from the old database to the new database and tune the database appropriately. You can use the following tools:
* [pgtune](https://pgtune.leopard.in.ua/#/)
* [postgresqltuner](https://github.com/jfcoz/postgresqltuner).
4. Turn of Zabbix Housekeeper in the Frontend as mentioned in the [Zabbix Partitioning Considerations](#zabbix-partitioning-considerations).
5. Shut down (or stop) the Zabbix Server and Zabbix Frontend from writing to the database.
6. Back up the original database!!!

### [Install PostgreSQL v11](#install-postgresql-v11)

Install PostgreSQL using the following commands on a Debian OS distro.

```bash
$ wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -
$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'
$ sudo apt update
$ sudo apt install postgresql-11
```

### [Prepare Zabbix Database](#prepare-zabbix-database)

Create a database user with permissions to create database objects. Ensure the created user (`zabbix`) is the owner of the `zabbix` database (`-O zabbix`).

```bash
$ sudo -u postgres createuser --pwprompt zabbix
$ sudo -u postgres createdb -O zabbix zabbix
```

Do not import the initial schema as creating the partitions will cause issues.

### [Create Empty history* and trends* Tables](#create-empty-history-and-trends-tables)

On an empty `zabbix` database (you can create multiple database on the same server if you'd like or want to upgrade from version 9.x to 10/11) create the following tables for `history*` and `trends*`.

First connecto the `zabbix` database as the `postgres` user with the following command

```bash
$ sudo -u zabbix psql zabbix
```

Then run the following SQL commands

```SQL
-- history
CREATE TABLE public.history
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    value numeric(16,4) NOT NULL DEFAULT 0.0000,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_1 ON public.history USING btree (itemid, clock);

-- history_log
CREATE TABLE public.history_log
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    "timestamp" integer NOT NULL DEFAULT 0,
    source character varying(64) COLLATE pg_catalog."default" NOT NULL DEFAULT ''::character varying,
    severity integer NOT NULL DEFAULT 0,
    value text COLLATE pg_catalog."default" NOT NULL DEFAULT ''::text,
    logeventid integer NOT NULL DEFAULT 0,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_log_1 ON public.history_log USING btree (itemid, clock);

-- history_str
CREATE TABLE public.history_str
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    value character varying(255) COLLATE pg_catalog."default" NOT NULL DEFAULT ''::character varying,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_str_1 ON public.history_str USING btree (itemid, clock);

-- history_text
CREATE TABLE public.history_text
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    value text COLLATE pg_catalog."default" NOT NULL DEFAULT ''::text,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_text_1 ON public.history_text USING btree (itemid, clock);

-- history_uint
CREATE TABLE public.history_uint
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    value numeric(20,0) NOT NULL DEFAULT (0)::numeric,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_uint_1 ON public.history_uint USING btree (itemid, clock);

-- trends
CREATE TABLE public.trends
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    num integer NOT NULL DEFAULT 0,
    value_min numeric(16,4) NOT NULL DEFAULT 0.0000,
    value_avg numeric(16,4) NOT NULL DEFAULT 0.0000,
    value_max numeric(16,4) NOT NULL DEFAULT 0.0000,
    CONSTRAINT trends_pkey PRIMARY KEY (itemid, clock)
) PARTITION BY RANGE (clock);

-- trends_uint
CREATE TABLE public.trends_uint
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    num integer NOT NULL DEFAULT 0,
    value_min numeric(20,0) NOT NULL DEFAULT (0)::numeric,
    value_avg numeric(20,0) NOT NULL DEFAULT (0)::numeric,
    value_max numeric(20,0) NOT NULL DEFAULT (0)::numeric,
    CONSTRAINT trends_uint_pkey PRIMARY KEY (itemid, clock)
) PARTITION BY RANGE (clock);
```

## [PostgreSQL Partition Manager Extension (pg_partman)](#postgresql-partition-manager-extension-pgpartman)

pg_partman is an extensions to create and manage both time-based and serial-based table partition sets. Native partitioning in PostgreSQL 10 is supported as of pg_partman v3.0.1 and PostgreSQL 11 as of pg_partman v4.0.0.

pg_partman works as an extension and it can be installed directly on top of PostgreSQL.

### [Installing pg_partman](#installing-pgpartman)

Debian apt:

```bash
$ sudo apt install postgresql-11-partman
```

### [postgresql.conf](#postgresqlconf)

Create the following configuration file in the proper postgresql directory:

```bash
$ vim /etc/postgresql/11/main/conf.d/pgpartman.conf
```

Add the following information there:

```conf
### General
shared_preload_libraries = 'pg_partman_bgw' # (this change requires restart)

### Partitioning & pg_partman settings
enable_partition_pruning = on
pg_partman_bgw.interval = 3600
pg_partman_bgw.role = 'zabbix'
pg_partman_bgw.dbname = 'zabbix'
pg_partman_bgw.analyze = off
pg_partman_bgw.jobmon = on
```

Restart postgresql (`sudo systemctl restart postgresql.service`) and in the logs you should see:

```logs
pg_partman master background worker master process initialized with role zabbix
```

Connect to the `zabbix` database and create the `pg_partman` extension as part of the `public` schema on the `zabbix` database. The following commands should be run by a **superuser** of the database cluster (i.e. 'postgres' is used in this example, suitable for Debian/Ubuntu).

```bash
$ sudo -u postgres psql zabbix
```

Then create the SCHEMA and EXTENSION:

```SQL
CREATE SCHEMA partman;
CREATE EXTENSION pg_partman schema partman;
```

Ensure the `zabbix` database user can execute all functions, procedures on the `partman` SCHEMA with the following SQL commands.
```SQL
GRANT ALL ON SCHEMA partman TO zabbix;
GRANT ALL ON ALL TABLES IN SCHEMA partman TO zabbix;
GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA partman TO zabbix;
GRANT EXECUTE ON ALL PROCEDURES IN SCHEMA partman TO zabbix;
```

### [Create Partitioned Tables](#create-partitioned-tables)

Perform the SQL queries **on the `zabbix` database as the `zabbix` user** (otherwise it won't find the functions in another database where the extension was added) which will target the pg_partman functions uploaded:

First connect to the `zabbix` database as the `zabbix` user:

```bash
$ sudo -u zabbix psql zabbix
```

Then run the following SQL commands:

```SQL
SELECT partman.create_parent('public.history', 'clock', 'native', 'daily', null, 7, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.history_uint', 'clock', 'native', 'daily', null, 7, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.history_str', 'clock', 'native', 'daily', null, 7, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.history_text', 'clock', 'native', 'daily', null, 7, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.history_log', 'clock', 'native', 'daily', null, 7, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.trends', 'clock', 'native', 'monthly', null, 12, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.trends_uint', 'clock', 'native', 'monthly', null, 12, 'on', null, true, 'seconds');
```

You should verify that the tables have ownership `zabbix`:

```SQL
\d
```

Example output:

```SQL
                    List of relations
 Schema |            Name            |   Type   | Owner
--------+----------------------------+----------+--------
 public | history                    | table    | zabbix
 public | history_default            | table    | zabbix
 public | history_log                | table    | zabbix
 public | history_log_default        | table    | zabbix
 public | history_log_p2019_08_24    | table    | zabbix
 public | history_log_p2019_08_25    | table    | zabbix
 public | history_log_p2019_08_26    | table    | zabbix
 public | history_log_p2019_08_27    | table    | zabbix
 public | history_log_p2019_08_28    | table    | zabbix
 public | history_log_p2019_08_29    | table    | zabbix
 public | history_log_p2019_08_30    | table    | zabbix
 public | history_log_p2019_08_31    | table    | zabbix
 public | history_log_p2019_09_01    | table    | zabbix
 public | history_log_p2019_09_02    | table    | zabbix
 public | history_log_p2019_09_03    | table    | zabbix
 public | history_log_p2019_09_04    | table    | zabbix
 public | history_log_p2019_09_05    | table    | zabbix
 public | history_log_p2019_09_06    | table    | zabbix
 public | history_log_p2019_09_07    | table    | zabbix
 public | history_p2019_08_24        | table    | zabbix
 public | history_p2019_08_25        | table    | zabbix
 public | history_p2019_08_26        | table    | zabbix
 public | history_p2019_08_27        | table    | zabbix
 public | history_p2019_08_28        | table    | zabbix
 public | history_p2019_08_29        | table    | zabbix
 public | history_p2019_08_30        | table    | zabbix
 public | history_p2019_08_31        | table    | zabbix
 public | history_p2019_09_01        | table    | zabbix
 public | history_p2019_09_02        | table    | zabbix
 public | history_p2019_09_03        | table    | zabbix
 public | history_p2019_09_04        | table    | zabbix
 public | history_p2019_09_05        | table    | zabbix
 public | history_p2019_09_06        | table    | zabbix
 public | history_p2019_09_07        | table    | zabbix
 public | history_str                | table    | zabbix
 public | history_str_default        | table    | zabbix
 public | history_str_p2019_08_24    | table    | zabbix
 public | history_str_p2019_08_25    | table    | zabbix
 public | history_str_p2019_08_26    | table    | zabbix
 public | history_str_p2019_08_27    | table    | zabbix
 public | history_str_p2019_08_28    | table    | zabbix
 public | history_str_p2019_08_29    | table    | zabbix
 public | history_str_p2019_08_30    | table    | zabbix
 public | history_str_p2019_08_31    | table    | zabbix
 public | history_str_p2019_09_01    | table    | zabbix
 public | history_str_p2019_09_02    | table    | zabbix
 public | history_str_p2019_09_03    | table    | zabbix
 public | history_str_p2019_09_04    | table    | zabbix
 public | history_str_p2019_09_05    | table    | zabbix
 public | history_str_p2019_09_06    | table    | zabbix
 public | history_str_p2019_09_07    | table    | zabbix
....
....
....
```

### [Deleting Designated Partitions](#deleting-designated-partitions)

It is impossible to manually remove partitions. Instead `UPDATE` the table `partman.part_config` table config.

Connect via `zabbix` user to the `zabbix` database:

```bash
$ sudo -u zabbix psql zabbix
```

Update the `partman.part_config` table:

```SQL
UPDATE partman.part_config set retention = '30 day', retention_keep_table = false, retention_keep_index = false WHERE parent_table = 'public.history';
```

Then execute maintenance procedure:

```SQL
SELECT partman.run_maintenance('public.history');
```

### [Partition Maintenance: Creating Future Partitions](#partition-maintenance-creating-future-partitions)

pg_partman has a function `run_maintenance` that allows one to automate the table maintenance.

```SQL
SELECT partman.run_maintenance(p_analyze := false);

 -- note: disabling analyze is recommended for native partitioning due to aggressive locks
```

**Native partitioning can result in heavy locking and therefore it is recommended to set p_analyze to FALSE which will effectively disable analyze.**

### [Partition Maintenance: Dropping/expiring old partitions](#partition-maintenance-droppingexpiring-old-partitions)

To configure pg_partman to drop old partitions, update the `partman.part_config` tables connected as the `zabbix` user:

```bash
$ sudo -u zabbix psql zabbix
```

Then run the following commands:

```SQL
UPDATE partman.part_config SET retention_keep_table = false, retention = '7 day'
WHERE parent_table = 'public.history';
UPDATE partman.part_config SET retention_keep_table = false, retention = '7 day'
WHERE parent_table = 'public.history_uint';
UPDATE partman.part_config SET retention_keep_table = false, retention = '7 day'
WHERE parent_table = 'public.history_str';
UPDATE partman.part_config SET retention_keep_table = false, retention = '7 day'
WHERE parent_table = 'public.history_text';
UPDATE partman.part_config SET retention_keep_table = false, retention = '7 day'
WHERE parent_table = 'public.history_log';
UPDATE partman.part_config SET retention_keep_table = false, retention = '12 month'
WHERE parent_table = 'public.trends';
UPDATE partman.part_config SET retention_keep_table = false, retention = '12 month'
WHERE parent_table = 'public.trends_uint';
```

Following this change run the maintenance actively via SQL:

```SQL
SELECT partman.run_maintenance('public.history');
SELECT partman.run_maintenance('public.history_uint');
SELECT partman.run_maintenance('public.history_str');
SELECT partman.run_maintenance('public.history_text');
SELECT partman.run_maintenance('public.history_log');
SELECT partman.run_maintenance('public.trends');
SELECT partman.run_maintenance('public.trends_uint');
```

---

## [Set up Zabbix PostgreSQL Partitioning on a Fresh Install](#set-up-zabbix-postgresql-partitioning-on-a-fresh-install)

To use partitioning on a fresh install on the following specifications (as an example):
* **Zabbix version 4.0**
* Debain 9 OS Distro
* PostgreSQL v.11

1. [Install PostgreSQL v11](#install-postgresql-v11).
2. [Prepare Zabbix Database](#prepare-zabbix-database).
3. [Create Empty history* and trends* Tables](#create-empty-history-and-trends-tables), [install pg_partman](#installing-pgpartman), ensure make the necessary changes in the [postgresql.conf file](#postgresqlconf), [create the partitioned tables](#create-partitioned-tables).  _Alternatively you can use my [ansible role to install pg_partman](https://github.com/Doctorbal/zabbix-postgres-partitioning/blob/master/ansible/zabbix.pgpartman) which performs the previous blocks_. Ensure you read the comments in the [`defaults/main.yml` file](https://github.com/Doctorbal/zabbix-postgres-partitioning/blob/master/ansible/zabbix.pgpartman/defaults/main.yml).
4. **Then import the Zabbix 4.0 schema by performing the following:**
    ```bash
    $ cd /tmp
    $ wget https://repo.zabbix.com/zabbix/4.0/debian/pool/main/z/zabbix-release/zabbix-release_4.0-3+stretch_all.deb
    $ sudo dpkg -i zabbix-release_4.0-3+stretch_all.deb
    $ sudo apt update
    $ sudo apt-get install --no-install-recommends zabbix-server-pgsql
    # Now import the schema. You will get errors stating some tables and indexes already exist but ignore those as they were purposefully set on the partitioning portion.
    $ zcat /usr/share/doc/zabbix-server-pgsql*/create.sql.gz | sudo -u zabbix psql zabbix
    ```
5. Continue with the rest of the installation of Zabbix (not listed here).

---

## [Troubleshooting](#troubleshooting)

### [How can I list all tables in partman schema?](#how-can-i-list-all-tables-in-partman-schema)

Run the following sql query in the `zabbix` DB as the `zabbix` user:

```sql
zabbix=# \dt partman.*
```

### [zabbix database user does not have appropriate permissions](#zabbix-database-user-does-not-have-appropriate-permissions)

Ensure that the `zabbix` user is the owner of all tables in the `zabbix` database.

You can use the [change-zbxdb-table-ownership.sh shell script](files/change-zbxdb-table-ownership.sh) to change the ownership of all zabbix tables easily.

```bash
$ sudo chmod +x /var/lib/postgresql/change-zbxdb-table-ownership.sh
$ sudo -u postgres /var/lib/postgresql/change-zbxdb-table-ownership.sh zabbix zabbix
```

### [importing the history table takes forever](#importing-the-history-table-takes-forever)

You should, generally speaking not retain `history` table data longer than a few days as the zabbix build-in historical data mechanism then kicks in; aka trends!

This is also suggested in the [zabbix documentation](https://www.zabbix.com/documentation/4.0/manual/config/items/history_and_trends). The `history` data (numeric float in this case) should be keep for the smallest possible number of days. I started using 7 days and find it plenty for my needs.

I would, for example, recommend to delete `history` data older than 14 days as a good starting point then tune your partitions as you see fit; there is no real need to store numeric (float) data longer than a few days as trends takes over. That way when you `pg_dump` and `pg_restore` the data it won't take a long time to filtrate into the appropriate partitions.

```SQL
delete FROM history where age(to_timestamp(history.clock)) > interval '14 days';
```

This thus makes the `pg_restore` using the `COPY` method much faster.

If you do intend to keep the `history` for a very long time you could disable the partman automatic maintenance via the following command:

```SQL
UPDATE partman.part_config SET automatic_maintenance = 'off' WHERE parent_table = 'public.history';
```

This is because importing a significant amount of data into the `history` partitioned table will take a very long time and running maintenance on that table will not work. Wait until the table is fully finished before turning on the maintenance for that table.

This was reported in [issue 5](https://github.com/Doctorbal/zabbix-postgres-partitioning/issues/5)

---

## [Change Zabbix history tables from monthly to daily with pgpartman](#change-zabbix-history-tables-from-monthly-to-daily-with-pgpartman)

* Originally I performed monthly partitions on the history* tables. But there is so much data being sent to Zabbix it increases the Zabbix database quickly. This will make scaling a problem in the future.
* Although I can keep attaching disks the simple, scalable and permanent solution is to implement "Daily" partitions on all history* tables.
* The Zabbix Consultants recommend using daily partitions no longer than 7 days previously. Additionally the Zabbix Official Documentation clarifies this in detail - https://www.zabbix.com/documentation/current/manual/config/items/history_and_trends
* The Zabbix Housekeeper, based on the rule set in Administration | General | Housekeeping to 7 days and 365 days respectively for history* and trends* data. Thus there is NO POINT IN RETAINING LONGER THAN 7 DAYS OF HISTORY IF ALREADY SPECIFIED IN THE FRONTEND.
* https://github.com/pgpartman/pg_partman/issues/248

**PRIOR**
1. Set 5 hour maintenance window. Take a snapshot of the database. pg_dump remotely the database and have backups upon backups...
2. Stop NGINX GUI, then Zabbix Server.
3. Kill all connections to the database.
4. Ensure all SQL commands are run as `zabbix` user (`$ sudo -u zabbix psql zabbix`).

```SQL
-- view connections
SELECT sum(numbackends) FROM pg_stat_database;

-- kill all connections
SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'zabbix'
  AND pid <> pg_backend_pid();
```

**PROCEDURE**

_Objective_: changing the following example table (plus 4 other tables ~10GGB each) from monthly to daily partitions while minimizing downtime and data loss.

```SQL
zabbix=# \d+ history
                                     Table "public.history"
 Column |     Type      | Collation | Nullable | Default | Storage | Stats target | Description
--------+---------------+-----------+----------+---------+---------+--------------+-------------
 itemid | bigint        |           | not null |         | plain   |              |
 clock  | integer       |           | not null | 0       | plain   |              |
 value  | numeric(16,4) |           | not null | 0.0000  | main    |              |
 ns     | integer       |           | not null | 0       | plain   |              |
Partition key: RANGE (clock)
Indexes:
    "history_1" btree (itemid, clock)
Partitions: history_p2019_01 FOR VALUES FROM<PHONE_NUMBER>) TO<PHONE_NUMBER>),
            history_p2019_02 FOR VALUES FROM<PHONE_NUMBER>) TO<PHONE_NUMBER>),
            history_p2019_03 FOR VALUES FROM<PHONE_NUMBER>) TO<PHONE_NUMBER>),
            history_p2019_04 FOR VALUES FROM<PHONE_NUMBER>) TO<PHONE_NUMBER>),
            history_p2019_05 FOR VALUES FROM<PHONE_NUMBER>) TO<PHONE_NUMBER>),
            history_p2019_06 FOR VALUES FROM<PHONE_NUMBER>) TO<PHONE_NUMBER>),
            history_p2019_07 FOR VALUES FROM<PHONE_NUMBER>) TO<PHONE_NUMBER>),
            history_default DEFAULT
```

Procedure:

0. Delete data older than 7 days. This will simplify and speed up the moving process.

```SQL
/* The following is just an example of an epoch timestamp 7 days out from initiall creating this procedure - https://www.epochconverter.com/ ... */
delete FROM history where age(to_timestamp(history.clock)) > interval '7 days';
delete FROM history_uint where age(to_timestamp(history_uint.clock)) > interval '7 days' ;
delete FROM history_str where age(to_timestamp(history_str.clock)) > interval '7 days' ;
delete FROM history_log where age(to_timestamp(history_log.clock)) > interval '7 days' ;
delete FROM history_text where age(to_timestamp(history_text.clock)) > interval '7 days' ;
```

1. Stop pg_partman from running the dynamic background worker to perform table maintenance on the `history*` tables in the `partman.part_config` column used by the maintenance

```SQL
UPDATE partman.part_config SET automatic_maintenance = 'off' WHERE parent_table = 'public.history';
UPDATE partman.part_config SET automatic_maintenance = 'off' WHERE parent_table = 'public.history_uint';
UPDATE partman.part_config SET automatic_maintenance = 'off' WHERE parent_table = 'public.history_str';
UPDATE partman.part_config SET automatic_maintenance = 'off' WHERE parent_table = 'public.history_log';
UPDATE partman.part_config SET automatic_maintenance = 'off' WHERE parent_table = 'public.history_text';
```

2. Create a table similar to the one being unpartitioned. For e.g.:

```SQL
-- history_moved
CREATE TABLE public.history_moved
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    value numeric(16,4) NOT NULL DEFAULT 0.0000,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_moved_1 ON public.history_moved USING BRIN (itemid, clock);

-- history_uint_moved
CREATE TABLE public.history_uint_moved
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    value numeric(20,0) NOT NULL DEFAULT (0)::numeric,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_uint_moved_1 ON public.history_uint_moved USING BRIN (itemid, clock);

-- history_str_moved
CREATE TABLE public.history_str_moved
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    value character varying(255) COLLATE pg_catalog."default" NOT NULL DEFAULT ''::character varying,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_str_moved_1 ON public.history_str_moved USING BRIN (itemid, clock);

-- history_log_moved
CREATE TABLE public.history_log_moved
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    "timestamp" integer NOT NULL DEFAULT 0,
    source character varying(64) COLLATE pg_catalog."default" NOT NULL DEFAULT ''::character varying,
    severity integer NOT NULL DEFAULT 0,
    value text COLLATE pg_catalog."default" NOT NULL DEFAULT ''::text,
    logeventid integer NOT NULL DEFAULT 0,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_log_moved_1 ON public.history_log_moved USING BRIN (itemid, clock);

-- history_text_moved
CREATE TABLE public.history_text_moved
(
    itemid bigint NOT NULL,
    clock integer NOT NULL DEFAULT 0,
    value text COLLATE pg_catalog."default" NOT NULL DEFAULT ''::text,
    ns integer NOT NULL DEFAULT 0
) PARTITION BY RANGE (clock);

CREATE INDEX history_text_moved_1 ON public.history_text_moved USING BRIN (itemid, clock);
```

Then run

```SQL
SELECT partman.create_parent('public.history_moved', 'clock', 'native', 'monthly', null, 1, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.history_uint_moved', 'clock', 'native', 'monthly', null, 1, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.history_str_moved', 'clock', 'native', 'monthly', null, 1, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.history_log_moved', 'clock', 'native', 'monthly', null, 1, 'on', null, true, 'seconds');
SELECT partman.create_parent('public.history_text_moved', 'clock', 'native', 'monthly', null, 1, 'on', null, true, 'seconds');
```

3. Call the `partman.undo_partition_proc()` function on the table wanting to be unpartitioned. This will insert the data into the newly created tables in the previous steps. *This seems to lock the table and you can't view any information in the frontend (hence a reason why the frontend should be stopped from writing to the DB)*:

```SQL
CALL partman.undo_partition_proc('public.history', '1 day', null, 1, 'public.history_moved', false, 0, 10, false);
CALL partman.undo_partition_proc('public.history_uint', '1 day', null, 3, 'public.history_uint_moved', false, 0, 10, false);
CALL partman.undo_partition_proc('public.history_str', '1 day', null, 3, 'public.history_str_moved', false, 0, 10, false);
CALL partman.undo_partition_proc('public.history_log', '1 day', null, 3, 'public.history_log_moved', false, 0, 10, false);
CALL partman.undo_partition_proc('public.history_text', '1 day', null, 3, 'public.history_text_moved', false, 0, 10, false);
```

```SQL
VACUUM ANALYZE history;
VACUUM ANALYZE history_uint;
VACUUM ANALYZE history_str;
VACUUM ANALYZE history_log;
VACUUM ANALYZE history_text;
```

4.  INSERT the data back into the newly partitioned tables. Use https://www.epochconverter.com/ to find epoch timestamp <= '7 days'.  Drop the old table and remove the `partman.part_config` column

```SQL
DROP TABLE history_moved;
DELETE FROM partman.part_config WHERE parent_table = 'public.history_moved';
DROP TABLE history_uint_moved;
DELETE FROM partman.part_config WHERE parent_table = 'public.history_uint_moved';
DROP TABLE history_str_moved;
DELETE FROM partman.part_config WHERE parent_table = 'public.history_str_moved';
DROP TABLE history_log_moved;
DELETE FROM partman.part_config WHERE parent_table = 'public.history_log_moved';
DROP TABLE history_text_moved;
DELETE FROM partman.part_config WHERE parent_table = 'public.history_text_moved';
```

7. UPDATE the `partman.part_config` for `public.history`:

```SQL
UPDATE partman.part_config SET automatic_maintenance = 'on', retention_keep_table = false, retention = '8 day' WHERE parent_table = 'public.history';
UPDATE partman.part_config SET automatic_maintenance = 'on', retention_keep_table = false, retention = '8 day' WHERE parent_table = 'public.history_uint';
UPDATE partman.part_config SET automatic_maintenance = 'on', retention_keep_table = false, retention = '8 day' WHERE parent_table = 'public.history_str';
UPDATE partman.part_config SET automatic_maintenance = 'on', retention_keep_table = false, retention = '8 day' WHERE parent_table = 'public.history_log';
UPDATE partman.part_config SET automatic_maintenance = 'on', retention_keep_table = false, retention = '8 day' WHERE parent_table = 'public.history_text';
```

8. Run the maintenance:

```SQL
SELECT partman.run_maintenance('public.history');
SELECT partman.run_maintenance('public.history_uint');
SELECT partman.run_maintenance('public.history_str');
SELECT partman.run_maintenance('public.history_log');
SELECT partman.run_maintenance('public.history_text');
```

9. Verify `partman.part_config`:

```SQL
SELECT * FROM partman.part_config;
```

10. Run VACUUM ANALYZE:

```SQL
VACUUM ANALYZE history;
VACUUM ANALYZE history_uint;
VACUUM ANALYZE history_str;
VACUUM ANALYZE history_log;
VACUUM ANALYZE history_text;
```

Ensure `pg_partman_bgw` is set in `postgresql.conf` file.

---

## [Zabbix Remote Data Dump](#zabbix-remote-data-dump)

### [pgdump/pgrestore Manual Mechanism](#pgdumppgrestore-manual-mechanism)

On the old database, ensure `pg_hba.conf` file is set to allow connections from the new database.

```bash
# Note that the following commands are run on the new DB instance...
$ sudo mkdir -p /var/backups/postgresql
$ sudo chown -R postgres:postgres /var/backups/postgresql
$ sudo -u postgres pg_dump -Fc --file=/var/backups/postgresql/zabbix.dump -d zabbix -h <OLD-`zabbix`-DATABASE> -U zabbix
$ sudo -u postgres pg_restore -j 8 -d zabbix /var/backups/postgresql/zabbix.dump -U zabbix
```

---

## [Upgrade Zabbix v3.4 to v4.2 while moving from PostgreSQL v9.6 to PostgreSQL v11](#upgrade-zabbix-v34-to-v42-while-moving-from-postgresql-v96-to-postgresql-v11)

The following scenario addresses how to migrate from Zabbix version 3.4 to version 4.2 while also moving from PostgreSQL version 9.6 to version 11.

**Requirements**
* Debian 8 Jessie or Debian 9 Stretch OS Distro
* Old PostgreSQL instance running version 9.6.
* New PostgreSQL instance running version 11.

Prepare the new PostgreSQL instance for the Zabbix 4.2 database as described in the [previous section](#prepare-postgresql-database), as well as [follow the steps for installing the pg_partman extension](#installing-pgpartman).

### [Migrate the Configuration Data](#migrate-the-configuration-data)

On the old database, ensure `pg_hba.conf` file is set to allow connections from the new database.

```bash
# Note that the following commands are run on the new DB instance...
$ sudo mkdir -p /var/backups/postgresql
$ sudo chown -R postgres:postgres /var/backups/postgresql
$ sudo -u postgres pg_dump -Fd -j 4 -d zabbix -h <OLD-`zabbix`-DATABASE> -U zabbix --inserts -Z 4 --file=/var/backups/postgresql/zabbix-configuration-<DATE>  --exclude-table=history* --exclude-table=trends*
$ sudo -u postgres pg_restore -Fd -j 4 -d zabbix -h <IP_ADDRESS> -U zabbix /var/backups/postgresql/zabbix-configuration-<DATE>
```

* In order to prevent the disk from trashing you can use the compression `pg_dump` option `-j 4`.
* Additionally using the custom directory format option `-Fd` helps restore data in parallel which is also faster.

At this point you will have all tables except `history*` and `trends*` on the new Zabbix instance.

You can then prepare the missing `history*` and `trends*` tables by following the ["create partitioned tables" section](#create-partitioned-tables) outlined above.

Once you finish the procedure of importing the configuration data from the old instance to the new instance **AND** you created the partitioned tables you can point the Zabbix server to the new DB instance.

Thus this minimizes downtime of the Zabbix environment as we can import the older `history*` and `trends*` data later while Zabbix is actively running.

Once you start the Zabbix Server you should see the following lines in the `zabbix_server.log` file verifying that the upgrade has completed successfully.

```logs
  6330:20190512:020451.749 using configuration file: /etc/zabbix/zabbix_server.conf
  6330:20190512:020451.862 current database version (mandatory/optional): 03040000/03040007
  6330:20190512:020451.862 required mandatory version: 04020000
  6330:20190512:020451.862 starting automatic database upgrade
  6330:20190512:020451.880 completed 0% of database upgrade
  6330:20190512:020451.897 completed 1% of database upgrade
  6330:20190512:020451.932 completed 2% of database upgrade
  ...
  6364:20190512:021416.145 completed 99% of event name update
  6364:20190512:021416.322 completed 100% of event name update
  6364:20190512:021416.555 event name update completed
  6364:20190512:021416.567 server #0 started [main process]
  ...
```

### [Migrate the old history* and trends* data](#migrate-the-old-history-and-trends-data)

In order to migrate the old data from `history*` and `trends*` data with as minimal time as possible perform the following `pg_dump/restore` command **on the new DB instance**:

Using the `--inserts` method for `history*` tables:

```bash
# Note that the following commands are run on the new DB instance...
$ sudo mkdir -p /var/backups/postgresql
$ sudo chown -R postgres:postgres /var/backups/postgresql
$ sudo -u postgres pg_dump -Fd -j 4 -d zabbix -h <OLD-`zabbix`-DATABASE> -U zabbix --inserts -Z 4 --file=/var/backups/postgresql/zabbix-history-tables-<DATE> --table=history*
$ sudo -u postgres pg_restore -Fd -j 4 -d zabbix -h <IP_ADDRESS> -U zabbix /var/backups/postgresql/zabbix-history-tables-<DATE>
```

You can use the default `COPY` procedure (not using the `--inserts` option...), as there are no constraints on the tables. Thus it is allowed to insert all the data, even the duplicate ones, to the `history*` tables. Those will eventually be dropped via partitioning.

Using the `COPY` method for `history*` tables:

```bash
# Note that the following commands are run on the new DB instance...
$ sudo mkdir -p /var/backups/postgresql
$ sudo chown -R postgres:postgres /var/backups/postgresql
$ sudo -u postgres pg_dump -Fc --file=/var/backups/postgresql/zabbix-history-tables-<DATE>.dump -d zabbix -h <OLD-`zabbix`-DATABASE> -U zabbix --table=history*
$ sudo -u postgres pg_restore -j 8 -d zabbix /var/backups/postgresql/zabbix-history-tables-<DATE>.dump -U zabbix
```

Using the `--inserts` method for `trends*` tables:

```bash
# Note that the following commands are run on the new DB instance...
$ sudo mkdir -p /var/backups/postgresql
$ sudo chown -R postgres:postgres /var/backups/postgresql
$ sudo -u postgres pg_dump -Fd -j 4 -d zabbix -h <OLD-`zabbix`-DATABASE> -U zabbix --inserts -Z 4 --file=/var/backups/postgresql/zabbix-trends-tables-<DATE> --table=trends*
$ sudo -u postgres pg_restore -Fd -j 4 -d zabbix -h <IP_ADDRESS> -U zabbix /var/backups/postgresql/zabbix-trends-tables-<DATE>
```

Using the `COPY` method for `trends*` tables:

```bash
# Note that the following commands are run on the new DB instance...
$ sudo mkdir -p /var/backups/postgresql
$ sudo chown -R postgres:postgres /var/backups/postgresql
$ sudo -u postgres pg_dump -Fc --file=/var/backups/postgresql/zabbix-trends-tables-<DATE>.dump -d zabbix -h <OLD-`zabbix`-DATABASE> -U zabbix --table=trends*
$ sudo -u postgres pg_restore -j 8 -d zabbix /var/backups/postgresql/zabbix-trends-tables-<DATE>.dump -U zabbix
```

For **large Zabbix databases** the amount of data to restore could be exponential (Terabytes worth of trends* data) and take a very long time (hours). A use case mentioned by someone else was to change the parameter in the `postgresql.conf` file `fsync = off` (:scream:) (this just requires a reload of postgresql and not a restart of the cluster). **This is very, very risky** as turning this off can cause unrecoverable data corruption. As an end result turning off _fsync_ helped 49 million records to restore within 3 hours.

:exclamation: As always please test this properly before implementing this procedure in a production instance.

---

## [Performance Testing](#benchmarking)

### [pgbench](#pgbench)

[`pgbench`](https://www.postgresql.org/docs/current/pgbench.html) help understand **Transaction Processing Performance Council (TPC)** on a database to see performance.

* Ensure you generate enough load.
* Options
    * `-i` = initialize
    * `-s` = scaling
    * `-j` = threads
    * `-c` = clients
    * `-t` = transactions; based on clients; if 10 clients then 100,000 transactions

Example:

```bash
$ pgbench -i -s 50 zabbix
$ pgbench -c 10 -j 2 -t 10000 zabbix
```

### [EXPLAIN ANALYZE](#explain-analyze)

To help understand what is being done under the hood with partitioning we can use the `EXPLAIN` command. The `EXPLAIN` command shows the execution plan of a statement and how the tables i scanned.

```bash
$ psql -d zabbix -c "EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) SELECT count(*) FROM public.history;"
```

```bash
$ psql -d zabbix -c "EXPLAIN ANALYZE SELECT * FROM public.history;"
```

### [Zabbix Databases Performance Results](#zabbix-database-performance-results)

**PSQL 11 Non-Partitioned Database VM**

* 8vCPU
* 16GB RAM
* SSD

```bash
$ pgbench -i -s 50 zabbix
$ pgbench -c 10 -j 2 -t 10000 zabbix
```

Result:

```bash
$ pgbench -c 10 -j 2 -t 10000 zabbix
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 10
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 100000/100000
latency average = 6.900 ms
tps = 1449.297191 (including connections establishing)
tps = 1449.363391 (excluding connections establishing)
```

**PSQL 11 Non-Partitioned Database Baremetal**

* 24 CPUs
* 126GB RAM
* HDD

```bash
$ pgbench -c 10 -j 2 -t 10000 zabbix
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 10
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 100000/100000
latency average = 8.235 ms
tps = 1214.341276 (including connections establishing)
tps = 1214.421294 (excluding connections establishing)
```

**PSQL 11 Partitioned Database VM**

* 8vCPU
* 16GB RAM
* SSD

```bash
$ pgbench -i -s 50 zabbix
$ pgbench -c 10 -j 2 -t 10000 zabbix
```

```bash
$ pgbench -c 10 -j 2 -t 10000 zabbix
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 10
number of threads: 2
number of transactions per client: 10000
number of transactions actually processed: 100000/100000
latency average = 5.605 ms
tps = 1783.971153 (including connections establishing)
tps = 1784.074779 (excluding connections establishing)
```

---

## [Partitioning Advantages, Mistakes and Problems](#partitioning-advantages-mistakes-and-problems)

### [Partitioning Advantages](#partitioning-advantages)

* Average number of index blocks to navigate in order to find a row goes down.
* Having smaller blocks of data will improve performance.
* You can `DROP` an individual partition to erase all of the data from that range.
* The `REINDEX` operations will happen in a fraction of a time it would take for a single giant index to build.

### [Common Partitioning Mistakes](#common-partitioning-mistakes)

* Not turning on `enable_partition_pruning = on`.
* Failing to add all the same indexes or constraints to each partition that existed in the parent.
* Forgetting to assign the same permissions to each child table as the parent.
* Writing queries that don't filter on the partitioned key field. The `WHERE` clause needs to filter on constants. In general, keep the `WHERE` clauses as simple as possible, to improve the odds the optimizer will construct the exclusion proof you're looking for.
* Query overhead for partitioning is proportional to the number of partitions. Keep the number of partitions to the two digit range for best performance.
* When you manually `VACUUM/ANALYZE`, these will not cascade from the parent. You need to specifically target each partition with those operations.
* Fail to account for out of range dates in the `INSERT` trigger. Expect the bad data will show up one day with a timestamp either far in the past or in the future, relative to what you have partitioned right now. Instead of throwing an error, some prefer to redirect inserted rows from outside of the partitioned range into a holding pen partition dedicated to suspicious data.
* Regarding **logical replication** note that the [logical replication restrictions from postgres documentation](https://www.postgresql.org/docs/current/logical-replication-restrictions.html) *Replication is only possible from base tables to base tables. That is, the tables on the publication and on the subscription side must be normal tables, not views, materialized views, partition root tables, or foreign tables. In the case of partitions, you can therefore replicate a partition hierarchy one-to-one, but you cannot currently replicate to a differently partitioned setup. Attempts to replicate tables other than base tables will result in an error.*

*Reference - PostgreSQL 10 High Performance; 2018; pg 426-427*

---

## [Ansible Role](#ansible-role)

I have written an ansible role that I used to configure pg_partman. I used the role in conjunction with the excellent [ANXS postgresql](https://github.com/ANXS/postgresql) role.

For details please view [ansible|zabbix.pgpartman](ansible/zabbix.pgpartman/).

---

## [References](#references)

* Ciolli, G., & Riggs, S. (2018). *PostgreSQL 10 Administration Cookbook.* Burmingham, UK: Packt Publishing Ltd.
* Ahmed, I., Pirozzi, E., Smith, G. (2018). *PostgreSQL High Performance.* Burmingham, UK: Packt Publishing Ltd.
* Juba, S., Volkov, A. (2017). *Learning PostgreSQL 10.* Burmingham, UK: Packt Publishing Ltd.
* [PostgreSQL Official Documentation | Table Partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html)
* [PostgreSQL Official Documentation | CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html)
* [PostgreSQL Official Documentation | Logical Replication | Restrictions](https://www.postgresql.org/docs/current/logical-replication-restrictions.html)
* [PostgreSQL Official Documentation | pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html)
* [PostgreSQL Official Documentation | Write Ahead Log](https://www.postgresql.org/docs/current/runtime-config-wal.html)
* [pgBarman](https://www.pgbarman.org/)
* [pgbench](https://www.postgresql.org/docs/current/pgbench.html)
* [pgBouncer](https://pgbouncer.github.io/)
* [PgCookbook - a PostgreSQL documentation project](https://github.com/grayhemp/pgcookbook/blob/master/switching_to_another_server_with_pgbouncer.md)
* [pglogical](https://www.2ndquadrant.com/en/resources/pglogical/)
* [pg_partman](https://github.com/pgpartman/pg_partman)
* [Upgrading to postgreSQL 11 with Logical Replication](https://blog.2ndquadrant.com/upgrading-to-postgresql-11-with-logical-replication/)
* [Zabbix Documentation | History and Trends](https://www.zabbix.com/documentation/current/manual/config/items/history_and_trends)
* [Tuning PostgreSQL with pgbench](https://blog.codeship.com/tuning-postgresql-with-pgbench/)
* [PostgreSQL Logical Replication Gotchas](https://pgdash.io/blog/postgres-replication-gotchas.html)
* [Change already implemented PSQL 11 Native Partitioning from Monthly to Daily](https://github.com/pgpartman/pg_partman/issues/248)
* [PostgreSQL Partition Manager Extension (`pg_partman`)](https://github.com/pgpartman/pg_partman/blob/master/doc/pg_partman.md)

repository: EvgenyEK/zabbix-postgres-partitioning

repo_name: EvgenyEK/zabbix-postgres-partitioning


text:
# POSH RTL Bridges reference design
Reference designs for mixed accuracy co-simulation using AXI, ACE, CHI, CXS
and PCIe hardware bridges.

## Before building

Make sure you've checked out the libsystemctlm-soc submodule:

```
$ git submodule update --init libsystemctlm-soc
```

Make sure you have Vivado 2018.3 installed and set up.

## For building the AXI bitstream

```
$ cd ./axi
$ make design TARGET=axi_cdma_axi4_128
```

## For building the ACE bitstream

```
$ cd ./ace
$ make
```

## For building the CHI bitstream

```
$ cd ./chi
$ make
```

## For building the CXS bitstream

```
$ cd ./cxs
$ make

```

## For building the PCIe bitstream

```
$ cd ./pcie
$ make pcie_ep
```

license: MIT

repository: edgarigl/posh-rtl-bridges-refdesigns

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: edgarigl/posh-rtl-bridges-refdesigns


text:
# Release Notes

See [https://www.unidata.ucar.edu/software/netcdf/documentation/NUG/](https://www.unidata.ucar.edu/software/netcdf/documentation/NUG/) for the rendered documentation.

## NetCDF User's Guide v1.2 - TBD

* [Organization] Renamed the directories to specify which is the `main` directory to work in, and `gradle` which we are slowly trying to move towards.

## NetCDF User's Guide v1.1 - April 16, 2021

* Migrated NUG from netCDF-C project.

repository: Unidata/netcdf

branch_name: refs/heads/main

gha_language: Makefile

gha_license_id: MIT

repo_name: Unidata/netcdf


text:
# common-compatible

common compatible tool

## install

`npm install common-compatible`

## document

1. Any kind of compatible functions or objects or anything may be added to this package.
2. There is no summary package file, such as index.js, in this package.
3. Require each single item to your file, or build you own compatible set.

## convention

### namespace

A file of compatible item should be placed in a classification folder.

### method convention

If a method has a `this` object, it should be defined as:

`function( thisObject, arg1, arg2, ... )`

### property convention

If a property has a `this` object, it's get/set method should be defined as:

get<Object><Property>= `function( thisObject )`
set<Object><Property>= `function( thisObject, value )`


text:
[![Netlify Status](https://api.netlify.com/api/v1/badges/8cf65901-dae4-4826-b32b-17feeebddc86/deploy-status)](https://app.netlify.com/sites/kind-montalcini-bed86a/deploys)

# Creating and updating the docs
Welcome to the GitHub repository for Volcano's public website. The docs are
hosted at https://volcano.sh.

We use [Hugo](https://gohugo.io/) to format and generate our website, and
[Netlify](https://www.netlify.com/) to manage the deployment of the site. Hugo
is an open-source static site generator that provides us with templates, content
organisation in a standard directory structure, and a website generation engine.
You write the pages in Markdown, and Hugo wraps them up into a website.

* Please see [How to contributing](https://github.com/volcano-sh/volcano/blob/master/contribute.md) for instructions on how to contribute, if you are not familiar with the
  GitHub workflow

## Quickstart

Here's a quick guide to updating the docs. It assumes you're familiar with the
GitHub workflow and you're happy to use the automated preview of your doc
updates:

1. Fork the repo on GitHub.
1. Make your changes and send a pull request (PR).
1. If you're not yet ready for a review, add a comment to the PR saying it's a
  work in progress or add `[WIP]` in your PRs title. You can also add `/hold` in a comment to mark the PR as not
  ready for merge. (**Don't** add the Hugo declarative "draft = true" to the
  page front matter, because that will prevent the auto-deployment of the
  content preview described in the next point.)
1. Wait for the automated PR workflow to do some checks. When it's ready,
  you should see a comment like this: **deploy/netlify — Deploy preview ready!**
1. Click **Details** to the right of "Deploy preview ready" to see a preview
  of your updates.
1. Continue updating your doc until you're happy with it.
1. When you're ready for a review, add a comment to the PR and assign a
  reviewer/approver.

## Previewing your changes on a local website server

If you'd like to preview your doc updates as you work, you can install Hugo
and run a local server. This section shows you how.

### Install Hugo

See the [Hugo installation guide][hugo-install]. Here are some examples:

#### Mac OS X:

```
brew install hugo
```

#### Debian:

1. Download the Debian package from the [Hugo website][hugo-install].
   Make sure to install the Hugo version specified by the `HUGO_VERSION` environment variable in the [`netlify.toml`](netlify.toml#L6) file.
   For example, [hugo_0.57.2_Linux-64bit.deb][hugo_0.57.2_Linux-64bit.deb].
1. Install the package using `dpkg`:

    ```
    sudo dpkg -i hugo_0.57.2_Linux-64bit.deb
    ```

1. Verify your installation:

    ```
    hugo version
    ```

### Run a local website server

Follow the usual GitHub workflow to fork the repo on GitHub and clone it to your
local machine, then use your local repo as input to your Hugo web server:

1.  Start your website server. Make sure you run this command from the
   `/website/` directory, so that Hugo can find the config files it needs:

    ```
    hugo server -D
    ```

1. Your website is at [http://localhost:1313/](http://localhost:1313/).

1. Continue with the usual GitHub workflow to edit files, commit them, push the
  changes up to your fork, and create a pull request. (See the GitHub workflow)

1. While making the changes, you can preview them on your local version of the
  website at [http://localhost:1313/](http://localhost:1313/). Note that if you
  have more than one local git branch, when you switch between git branches the
  local website reflects the files in the current branch.

Useful Hugo docs:
- [Hugo installation guide](https://gohugo.io/getting-started/installing/)
- [Hugo basic usage](https://gohugo.io/getting-started/usage/)
- [Hugo site directory structure](https://gohugo.io/getting-started/directory-structure/)
- [hugo server reference](https://gohugo.io/commands/hugo_server/)
- [hugo new reference](https://gohugo.io/commands/hugo_new/)

[hugo-install]: https://gohugo.io/getting-started/installing/
[hugo_0.57.2_Linux-64bit.deb]: https://github.com/gohugoio/hugo/releases/download/v0.57.2/hugo_0.57.2_Linux-64bit.deb

repository: volcano-sh/website

gha_language: HTML

repo_name: volcano-sh/website


text:
Employee Tracker
========

by Debbie Cravey and Jessica Hori

Employee Tracker is a simple directory of a company's divisions and employees using Active Record to control SQL database functions.

Installation
------------

Install Employee Tracker by first cloning the repository.

```
$ git clone https://github.com/dacravey/employee_tracker.git
```

Install all of the required gems:
* ('pg')
* ('sinatra')
* ('sinatra-contrib')
* ('pry')
* ('rspec')
* ('sinatra-activerecord')
* ('rake')
```
$ bundle install
```
Start the database:
```
$ postgres
```

Create the databases and tables by running the following:
```
$ rake db:create
$ rake db:migrate
$ rake db:test:prepare
```

Start the webserver:
```
$ ruby app.rb
```

In your web browser, go to http://localhost:4567

Contribute
----------
- Issue Tracker: github.com/dacravey/employee_tracker/issues
- Source Code: https://github.com/dacravey/employee_tracker.git

Support
-------

If you are having issues, please let me know at<EMAIL_ADDRESS>
Bug reports
===========

If you discover any bugs, feel free to create an issue on GitHub. Please add as much information as possible to help with fixing the possible bug. I also welcome your help if you choose to fork and send a pull request.

https://github.com/dacravey/repository/issues

License
=======

MIT License. Copyright 2015 Debbie Cravey & Jessica Hori


text:
Magikarp-Popups
===============

Easily create modern-looking popups with a Title, Prompt and up to 8 different buttons, to which you can define a function for each.

Getting Started
===============

Load magikarp.popups.js file into your document, along with jquery and jquery ui:
  ```javascript
  <script src="magikarp.popups.js"></script>
  <script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
  <script src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.4/jquery-ui.min.js"></script>
  ```

Add the "data-social-popup" expando to any html element, it must contain the following (Each separated by ";"):

Headline

Prompt

Buttons (up to 8)

      :btn: prefix.

      Caption

      Background Color (Foreground will always be #fff)

      Function Name

For example:

```html
<img src="http://www.pentcloud.com/social/app/images/addTask.png" data-social-popup="Eliminar Contacto;¿Estás seguro de querer eliminar este contacto?;:btn:Si-#f9ae41-deleteContact:btn:No-#ef3e42-SocialPopupsHide">
```

When clicked would create the following result:

So that when we click on "Si" it will call deleteContact() and when we click on "No" it will call SocialPopupsHide().

SocialPopupsHide() is a built-in function that will destroy the popup after used, you might include it in your own callback functions as well as a default for cancel buttons.

repository: diegofcornejo/Magikarp-Popups

repo_name: diegofcornejo/Magikarp-Popups


text:
# Contao Developer Bundle
Helper for Contao 4 Open Source CMS

The following functions are added by this bundle:
- New "Start- / Stop-Wrapper" Content-Element
- Adds an insert tag to receive GET-Paremters `{{get::*}}`
- Overwrites the mod_article template and inserts an inner wrapper (the default wrapper remains available via the template "mod_article_default")

repository: oveleon/contao-developer-bundle

repo_name: oveleon/contao-developer-bundle


text:
sbt-js: SBT Javascript Plugin
=============================

[Simple Build Tool] plugin for compiling Javascript files from multiple sources using
Google's [Closure compiler].

Copyright (c) 2011 [Dave Gurnell] of [Untyped].

[Simple Build Tool]: http://simple-build-tool.googlecode.com
[Closure compiler]: http://code.google.com/p/closure-compiler
[Dave Gurnell]: http://boxandarrow.com
[Untyped]: http://untyped.com

Installation
============

For SBT 0.11:

Create a `project/plugins.sbt` file and paste the following content into it:

    resolvers += "Untyped Public Repo" at "http://repo.untyped.com"

    addSbtPlugin("untyped" %% "sbt-js" % "0.1-SNAPSHOT")

Then, in your build.sbt file, put:

    seq(jsSettings : _*)

If you're using [xsbt-web-plugin](https://github.com/siasia/xsbt-web-plugin "xsbt-web-plugin"),
add the output files to the webapp with:

    (webappResources in Compile) <+= (resourceManaged in Compile)

To change the directory that is scanned, use:

    (sourceDirectory in (Compile, JsKeys.js)) <<= (sourceDirectory in Compile)(_ / "path" / "to" / "js-files")

To cause the `js` task to run automatically when you run `compile`:

    (compile in Compile) <<= compile in Compile dependsOn (JsKeys.js in Compile)

To use pretty-printing instead of regular Javascript minification:

    (JsKeys.prettyPrint in (Compile, JsKeys.js)) := true

To use more aggressive variable renaming (producing smaller output files that are less
likely to work without care):

    (JsKeys.variableRenamingPolicy in (Compile, JsKeys.js)) := VariableRenamingPolicy.ALL

Or to turn variable renaming off altogether:

    (JsKeys.variableRenamingPolicy in (Compile, JsKeys.js)) := VariableRenamingPolicy.NONE

Usage
=====

To compile Javascript sources, use the `js` command in sbt. Read the installation
instructions above to see how to include Javascript compilation as part of the regular
`compile` command.

The default behaviour of the plugin is to scan your `src/main` directory and look for
two types of files:

 - Javascript files, with extension `.js`
 - Javascript manifest files, with extension `.jsm` or `.jsmanifest`

These files are compiled and minified using Google's Closure compiler and placed in
equivalent locations under `target/scala-2.9.x/resource_managed`.

Read on for a description of the handling for each type of file.

### *Require* statements in Javascript files

You can add *require* statements to your Javascript files to specify dependencies
on other files or URLs. Require statements can be of the following forms:

    // require "path/to/my/file.js"

    // require "http://mywebsite.com/path/to/my/file.js"

Required files are *prepended* to the file they appear in, and the whole lot is
passed through the Google Closure compiler for minification. Note the following:

 - paths are resolved relative to the file they appear in;

 - the position of a require statement in the source does not matter - dependencies
   are always inserted just before the beginning of the file;

 - if multiple require statements are present in a file, the dependencies are inlined
   in the order they are required;

 - dependencies can be recursive - files can require files that require files;

 - woe betide you if you create recursive dependencies between your files :)

### Javascript manifest files

Javascript manifest files are a useful shorthand for building Javascript from a list
of sources. A manifest contains an ordered list of JavaScript source locations.
For example:

    # You can specify remote files using URLs...
    http://code.jquery.com/jquery-1.5.1.js

    # ...and local files using regular paths
    #    (relative to the location of the manifest):
    lib/foo.js
    bar.js

    # Blank lines and bash-style comments are also supported.
    # These may be swapped for JS-style comments in the future.

The sources are cached and inlined into one large Javascript file, which is then
passed to the Closure compiler. The compiler outputs a file of the same name and
relative path of the manifest, but with a `.js` extension. For example, if your
manifest file is at `src/main/javascript/static/js/kitchen-sink.jsm` in the source
tree, the final path would be `resource_managed/main/static/js/kitchen-sink.js`
in the target tree.

Templating
==========

It is sometime useful to template Javascript files. For example, you might want
scripts to refer to `localhost` during development and your live server once deployed.

Javascript files with the extension `.template.js` are passed through a [Mustache]
template processor before being passed to the Closure compiler.

Property names and values are drawn from a properties file that is located and parsed
in an identical manner to the Lift web framework (though the implementation has no
dependency on Lift). The default location for property files is `src/main/resources/props`.
See the [Lift documentation] for file formats and naming conventions.

[Mustache]: http://mustache.github.com/
[Lift documentation]: http://www.assembla.com/spaces/liftweb/wiki/Properties

Acknowledgements
================

v0.6+ for SBT 0.11 based on [less-sbt](https://github.com/softprops/less-sbt), Copyright (c) 2011 Doug Tangren.
v0.1-v0.5 for SBT 0.7 based on [Coffee Script SBT plugin], Copyright (c) 2010 Luke Amdor.

Heavily influenced by the [YUI Compressor SBT plugin] by Jon Hoffman.

Thanks to:

 - [Tim Nelson](https://github.com/eltimn) for his work on the SBT 0.11
   migration and dramatic improvements to this README.

 - [Glade Diviney](https://github.com/gladed) for help producing test cases
   and debugging various issues.

[Coffee Script SBT plugin]: https://github.com/rubbish/coffee-script-sbt-plugin
[YUI Compressor SBT plugin]: https://github.com/hoffrocket/sbt-yui

Licence
=======

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

repository: eltimn/sbt-plugins

extension: markdown

repo_name: eltimn/sbt-plugins


text:
---
title: PlantUML activity diagram that has a lot of rules
tags: plantuml uml
---
## PlantUML activity diagram that has a lot of rules

## Question

Create a PlantUML activity diagram for the following flow:

- Upload CSV file
- Check if the user is already enabled. If yes, stop.
- If no, check if user count(SSO id) > 1 based on bCode and userId. If yes, mark user has issue1 and stop.
- If no, check if user count(SSO id) = 0 based on bCode and userId. If yes, mark user ready and stop.
- If no, check if user count(SSO id) > 1 based on email. If yes, mark user has issue2 and stop.
- If no, check if user count(bCode and userId) > 1 based on email. If yes, mark user has issue3 and stop.
- If no, check if user count(bCode and userId) = 1 and count(SSO id) = 1 based on email. If yes, mark user already enabled and insert user to friendly and stop.
- If no, stop

## Answer

```plantuml
@startuml
start
:Upload CSV file;
if (Is user already enabled?) then (yes)
  stop
else (no)
  if (User count(SSO id) > 1 based on bCode and userId?) then (yes)
    :Mark user has issue1;
    stop
  else (no)
    if (User count(SSO id) = 0 based on bCode and userId?) then (yes)
      :Mark user ready;
      stop
    else (no)
      if (User count(SSO id) > 1 based on email?) then (yes)
        :Mark user has issue2;
        stop
      else (no)
        if (User count(bCode and userId) > 1 based on email?) then (yes)
          :Mark user has issue3;
          stop
        else (no)
          if (User count(bCode and userId) = 1 and count(SSO id) = 1 based on email?) then (yes)
            :Mark user already enabled;
            :Insert user to friendly;
            stop
          else (no)
            stop
          endif
        endif
      endif
    endif
  endif
endif
@enduml
```

The diagram starts with the "Upload CSV file" activity and then goes through a series of checks to determine the user's status. If the user is already enabled, the flow stops. Otherwise, it checks for various conditions such as the number of users with the same SSO ID or email address. If any of these conditions are met, the user is marked with an issue and the flow stops. If all checks are passed, the user is marked as ready or already enabled, and in the latter case, the user is inserted into the friendly list. Finally, the flow ends with the "stop" activity.


text:
# Installation
> `npm install --save @types/semver`

# Summary
This package contains type definitions for semver (https://github.com/npm/node-semver).

# Details
Files were exported from https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/semver

Additional Details
 * Last updated: Fri, 25 Oct 2019 20:12:50 GMT
 * Dependencies: none
 * Global values: none

# Credits
These definitions were written by Bart van der Schoor <https://github.com/Bartvds>, BendingBender <https://github.com/BendingBender>, Lucian Buzzo <https://github.com/LucianBuzzo>, Klaus Meinhardt <https://github.com/ajafff>, and ExE Boss <https://github.com/ExE-Boss>.

repository: stCarolas/setup-maven

gha_language: JavaScript

repo_name: stCarolas/setup-maven


text:
---
layout: layout.pug
navigationTitle:  dcos nifi pod restart
title: dcos nifi pod restart
menuWeight: 4
excerpt: Restart a given pod without moving it to a new agent.
enterprise: false
model: ../../../../data.yml
render: mustache
---

# Description
The `dcos nifi pod restart` command restarts a given pod without moving it to a new agent.

# Usage

```bash
dcos nifi pod restart <pod> [flags]
```

# Options

| Name |  Description |
|---------|-------------|
| `-h, --help`   |  Show context-sensitive help. |
| `-v, --verbose`   |  Enable extra logging of requests/responses |
| `--name="nifi"`   |  Name of the service instance to query |

# Parent command

| Command | Description |
|---------|-------------|
| [dcos nifi pod](../) | Displays pod related informations and allows pod operations of the NiFi service. |

repository: mesosphere/dcos-docs-site

gha_language: SCSS

repo_name: mesosphere/dcos-docs-site


text:
# ansible-minecraft
Install Minecraft on Debian Jessie.

- To build a Minecraft Docker container see `docker-builder/README.md`.
- To deploy a Minecraft Docker container see `docker-deploy/README.md`.

To install Minecraft without Docker, continue :).

Requirements
------------
* [Ansible](http://docs.ansible.com/ansible/intro_installation.html).
* [Vagrant](https://www.vagrantup.com/) (If you want to build the Docker container).
  * [VirtualBox](https://www.virtualbox.org/wiki/Downloads) or [vmware](http://www.vmware.com/).
* A Debian Jessie server (May work on Ubuntu).

- You will need to add your server's IP to the `inventories` file.

How to run
----------
Example ansible command: `ansible-playbook -i inventories playbook.yml -u <user>`.

Configurable options
--------------------
To see all available configurable options and their defaults see `roles/minecraft/defaults/main.yml`.

Example Playbook:

```

- hosts: minecraft
  become: yes
  vars:
    update_server: yes
    agree_to_eula: "true" #Do you agree to the eula? See https://account.mojang.com/documents/minecraft_eula
    install_location: "/home/minecraft"
    minecraft_version: 1.9
    minecraft_url: "https://s3.amazonaws.com/Minecraft.Download/versions/{{ minecraft_version }}/minecraft_server.{{ minecraft_version }}.jar"
    minecraft_sha256sum: "38a797f50c71f55202e2135a30302cf3a5c8cb494c6d225b88599542957d3a7d"
    java_options: "-Xms2048M -Xmx2048M -jar {{ install_location }}/minecraft_server.{{ minecraft_version }}.jar nogui" #Change the Xms and Xmx values base on the resources of your server.
    white_list: "true"
    white_list_include:
      - username1
      - username2
  roles:
    - minecraft

```

Acknowledgements
----------------
- Used directions off of a gamepedia Minecraft [Tutorial](http://minecraft.gamepedia.com/Tutorials/Setting_up_a_server)

Things to Consider
------------------
As of right now I am not including any sort of Firewall rules (Such as iptables). Before installing and running any server(particularly minecraft)
you should understand the implications.

repository: CorbanR/ansible-minecraft

repo_name: CorbanR/ansible-minecraft


text:
---
layout: post-for-papers
title:  "Deep learning and process understanding for data-driven Earth system science"
date:   2020-02-17 19:05:26 +0800
excerpt: "Machine learning approaches are increasingly used to extract patterns and insights from the ever-increasing stream of geospatial data, but current approaches may not be optimal when system behaviour is dominated by spatial or temporal context. Here, rather than amending classical machine learning, we argue that these contextual cues should be used as part of deep learning (an approach that is able to extract spatio-temporal features automatically) to gain further process understanding of Earth system science problems, improving the predictive ability of seasonal forecasting and modelling of long-range spatial connections across multiple timescales, for example. The next step will be a hybrid modelling approach, coupling physical process models with the versatility of data-driven machine learning."
papers: true
---

关于DL应用于地球科学的文章：

* [Deep learning for multi-year ENSO forecasts](https://www.nature.com/articles/s41586-019-1559-7)
* [AI COPERNICUS ‘DISCOVERS’ THAT EARTH ORBITS THE SUN](https://europepmc.org/abstract/med/31719694)
* [Complex networks reveal global pattern of extreme-rainfall teleconnections ](https://www.nature.com/articles/s41586-018-0872-x) (还没想好放到那个目录，姑且当作是深度学习的引用)
* []()
* []()
* []()

[Deep learning and process understanding for data-driven Earth system science](https://www.nature.com/articles/s41586-019-0912-1) is a review published in [《Nature》](https://www.nature.com/articles/s41586-019-0912-1) as a [Perspectives](https://www.nature.com/nature/articles?type=perspective)

In the Paper, author review the development of machine learning in the geoscientific context, and highlight how deep learning—that is, the automatic extraction of abstract (spatio-temporal) features—has the potential to overcome many of the limitations that have, until now, hindered a more wide-spread adoption of machine learning. Author further lay out the most promising but also challenging approaches in combining machine learning with physical modelling.

## Big data challenges

![Big data challenges in the geoscientific context.](https://github.com/Clynie/Clynie.github.io/blob/master/_img/Big_data_challenges_in_the_geoscientific.png?raw=true)

首先文章介绍了地球系统的资料属于标准的 **“Four V's”** of __“Big Data”__:__Volume__(深度), __velocity__(更新速度), __Variety__(多样性), __Veracity__(真实性)。

紧接着文章提到了两个：“事实”：
1. 我们收集、创造数据的能力远远的超过我们能够明智的选择数据的能力，更别说理解它们。
2. 在过去几十年中，预测能力并没有随着数据的可用性的增加而增加。

## Four examples of typical deep learning applications

![Four examples of typical deep learning applications.](https://github.com/Clynie/Clynie.github.io/blob/master/_img/Four examples of typical deep learning applications.png?raw=true)

作者介绍了深度学习的四个引用例子：

* [Object recognition](https://github.com/eracah/hur-detect.git) 对应极端事件识别
* Super-resolution applications 对应分辨率处理
* Video prediction 对应短临预测
* Language translation 对应时间序列预测

我是个初学者，对于上面作者提到的四个例子，我个人觉得其实主要就是一个是CNN和Sequence Model的应用。
比较容易的是可以采用[YOLO](https://pjreddie.com/darknet/yolo/)

## Integration with physical modelling

在模式方面，一般来说物理模式和机器学习被认为是两种不同领域；一个是根据理论，一个是根据数据。
但是这两个方式其实应该是互补的；因此作者在这里阐述了如何将数据科学应用到理论科学中。

![Linkages between physic    al models and machine learning.](https://github.com/Clynie/Clynie.github.io/blob/master/_img/Linkages between physical models and machine learning.jpg?raw=true)

![Interpretation of hybrid modelling](https://github.com/Clynie/Clynie.github.io/blob/master/_img/Interpretation of hybrid modelling.jpg?raw=true)


text:
# poutils

<!-- vi:se ts=4 sts=4 tw=78 et ai: -->

Osamu Aoki<EMAIL_ADDRESS>
This`poutils` provides generic helper programs and utility class to play with
PO files: https://github.com/osamuaoki/poutils

Reverse i18n workflow tools:

* `po_align`: Align msgstr and msgid candidate pairs to the original file
  location with duplicate entries using po4a generated files.  -- align PO
  sequence before po_combine
* `po_combine`: generate a PO file from POT files from the master
  and translation data. --  reverse i18n workflow.  see "po_combine -h"
* `po_clean`: Clean up msgstr matched msgid intelligently (excluding
  <screen>, http...).  -- unset untranslated msgstr intelligently
* `po_check`: check matching between msgid and msgstr in a merged PO file.
  -- sanity check on msgid and msgstr contents.  see "po_check -h"
* `po_rm_fuzzy`: Remove fuzzy from all PO contents -- remove fuzzy flags in PO

GUI/CUI PO editor helper tools: (previous related)

* `po_update`: Update msgstr matching with the previous msgid with the
  updated msgid.-- update not-for-translation msgstr intelligently
* `po_wdiff`: Convert the previous msgid data into the wdiff-like data
   -- easy identification of the upstream changes
* `po_previous`: Revert changes made by `po_wdiff`

## Development of this package

### Git repo usage

* *master* : upstream development with non-native version number
  (This works since I am upstream and maintainer.  See "man dgit-maint-merge")

### Package build

    $ git deborig -f HEAD
    $ debuild

or

    $ ./setup.py deb

repository: osamuaoki/poutils

repo_name: osamuaoki/poutils


text:
[![Unit Tests](https://github.com/slightlynybbled/engineering_notation/actions/workflows/unittest.yml/badge.svg)](https://github.com/slightlynybbled/engineering_notation/actions/workflows/unittest.yml)

# Purpose

To easily work with human-readable engineering notation.  I wrote this as a quick tool for my own use.
I found that I was writing the same functionality into multiple packages and would like a quick pip-installable
package to take care of this manipulation for me.  The package should be easily extended for other use cases.
The package is unit-less, so only operates on numeric values.  Unit detection may be added in future versions.

More information may be found at [for(embed)](http://forembed.com/engineering-notation-in-python.html).

# Installation

Install using pip: `pip install engineering_notation`.

# Status and Contributions

This project currently has 100% test coverage.  Have a look in `test.py` for examples of how to use
this library. To execute the tests, run `pytest` from the main directory or,
in some environments, it may be necessary to run `python3 -m pytest`.

Any contributions must pass 100% of current tests and pass flake8.  To execute
flake8, navigate to the project directory and `python3 setup.py flake8`.

Your pull request will automatically be run through testing and flake8 checks and
any pull requests that do not pass these will be put on hold pending passing.

# Use

There are multiple ways of initializing a number to a particular value, but a string is the preferred method:

```
>>> from engineering_notation import EngNumber
>>> EngNumber('10k')
10k
>>> EngNumber('10000')
10k
>>> EngNumber(10000)
10k
>>> EngNumber(10000.0)
10k
>>> EngNumber(1e4)
10k
```

Where decimals are involved, we use a default precision of 2 digits:

```
>>> EngNumber('4.99k')
4.99k
>>> EngNumber('4.9k')
4.90k
```

This behavior can truncate your results in some cases, and cause your number to round.  To specify more or less
digits, simply specify the precision in the declaration:

```
>>> EngNumber('4.999k')
5k
>>> EngNumber('4.999k', precision=3)
4.999k
```

Most operations that you would perform on numeric values are valid, although all operations are not implemented:

```
>>> EngNumber('2.2k') * 2
4.40k
>>> 2 * EngNumber('2.2k')
4.40k
>>> EngNumber(1.2) > EngNumber('3.3k')
False
>>> EngNumber(1.2) <= EngNumber('3.3k')
True
>>> EngNumber('3.3k') == EngNumber(3300)
True
```

All of the above operations are also possible on the `EngUnit()` class as well.  The only difference is
that units must match for addition/subtraction/comparison operations.  Although multiplication and division
operations will work numerically, they may not always be strictly correct.  This is because EngUnit is not
intended to replace a computer algebra system!

```
>>> EngUnit('2s') / EngUnit('4rotations')
0.5s/rotations
```

Additionally, since there are 'reserved' letters for sizing the number, you must be careful with your units!

```
>>> EngUnit('2mm')
2mm        # <<< this value equivalent to "0.002m"
>>> EngUnit('2meter')
2meter     # <<< this value is equivalent to "0.002eter", the "m" was used to scale the unit!
>>> EngUnit('2', unit='meter')   # <<< this will work better
```

# Contributions

Contributions are welcome.  Feel free to make feature requests in the issues.

gha_language: Python


text:
# CMS with Laravel + Angular Material starter project
> Version 1.0

![Angular & Laravel](https://raw.githubusercontent.com/henrychavez/cms-laravel-angular-material/master/public/img/logo.png)

## OVERVIEW
This is a starter project for a CMS where you cad administrate your app or website.

* Laravel 5.1
* Angular
* Angular Material v10.0
* Blazing fast Elixir 3.0 configuration with custom tasks
* Artisan generators for angular (artisan ng:view name, etc..)
* Based on Angular Style Guide by @john_papa

## Installation

    composer create-project bytpher/cms-laravel-angular-material [project name] --prefer-dist
    cd [project name]
    #fix database credentials in .env
    npm install || sudo npm install for Mac users
    gulp && gulp watch
    php artisan serve

* You're ready to go! http://localhost:8000
* More Features and documentation coming soon.


text:
# Acceptance Tests

## Overview

This project contains the acceptance tests that you can run as part of Kyma testing process.
The tests are written in Go. Each component or group of scenarios has a separate folder, like `dex` or `servicecatalog`.

## Usage
This section provides information on building and versioning of the Docker image, as well as configuring Kyma.

### Adding new tests
1. Add a new package
2. Modify the Dockerfile to compile the test binary to pkg.test
3. Add execution of the test to the `entrypoint.sh` script

### Configuring Kyma

After building and pushing the Docker image, set the proper tag in the `resources/core/values.yaml` file, in the`acceptanceTest.imageTag` property.

repository: Disper/kyma

repo_name: Disper/kyma


text:
---
title: Internet Explorer 8 Support Will Be Removed in Ember 2.0
authors:
  - tom-dale
date: 2015-04-20T00:00:00.000Z
tags:
  - compatibility
  - '2015'
---

After reviewing [this thread][rfc-ie8], it seems clear that the vast majority of
Ember users who have responded, including people working at large
corporations, are comfortable with dropping IE8 support in Ember 2.0.
On the other hand, while there is enormous support for dropping IE9
support as well, a number of people still rely on support for IE9, and
the benefits of dropping IE9 in Ember 2.0 are not as strong.

After reviewing discussion on the RFC proposal, many in-person conversations with Ember
users in large companies, and reviewing the private data sent to us via
email, we have decided that Ember 2.0 will support IE9+.

So how are we going to manage this transition, and what should you do
if your business still requires IE8 support for the time being?

## 1.13 with Extended Browser Support

The core team will continue to periodically release point releases in
the 1.13 series to patch security bugs and browser compatibility
issues, including issues in IE8.

No new features will be added, and we should be clear that we do not
intend people to stay on this release unless they must support IE8. Our
Semantic Version guarantees mean that the vast majority of the
community should migrate to the 2.x series as soon as possible.

It is important to note that Ember 1.13 will come with deprecation
warnings for everything that we will break in Ember 2.0. As a result,
if you are running Ember 1.13 without any deprecation warnings, you
should be able to upgrade to Ember 2.0. And because of the
Semantic Versioning guarantees in the Ember 2.x series, you may only have
few issues when upgrading from Ember 1.13 to the most recent version
of Ember 2.x when you are able to drop IE8 support.

For example, imagine you build the Ember app for Big Widget Enterprise
Co. that requires IE8 support. You upgrade to 1.13 (the last release in
the 1.x series) and, over time, refactor code to eliminate all
deprecation warnings. Periodically, you apply 1.13 patch releases to
maintain browser compatibility and to fix potential security issues.

Then, in April of 2016, management decides that enough customers have
moved off IE8 that you no longer need to support it. At that time,
Ember 2.6 will be the most recent stable release. Because 1.13 without
deprecation warnings is forwards-compatible with Ember 2.6, you can
upgrade from 1.13 to 2.6 with little hassle.

With the integration of Ember CLI and Ember Data into the Semantic
Versioning guarantees, many of your dependencies will follow a similar
upgrade path.

## Ecosystem

The above guarantees only apply to Ember, Ember Data, Ember
CLI, and the rest of the core-supported packages. Addon authors are
free to define their own support matrices. We encourage those who
depend on older browsers to contribute back by submitting PRs to the
addons they use with compatibility patches. Likewise, we encourage
authors of existing addons to work with users to offer a browser
compatibility matrix as close to the core projects as possible.

If you require support for IE8 (and as a result, Ember 1.13), make sure
to make your voice heard across the addon ecosystem.

That said, you should expect that new addons that come out after Ember
2.0 will not target Ember 1.13, and you should factor that into your
decision to remain on the 1.13 Extended Browser Support release of
Ember.

## FastBoot

FastBoot, our effort to bring server-side rendering to all Ember apps,
is designed to offer even users with slow, low-feature browsers a fast
experience. While most people think of this as a benefit to mobile
users, IE8 certainly qualifies as a slow, low-feature browser.

Because Ember applications are written "route first", any idiomatic
Ember content app that uses links as the primary mode of navigation
will be able to provide a passable experience for users with an
unsupported version of JavaScript, or no JavaScript at all.

It is worth noting that FastBoot, in the medium term, will have good
support for read-only content sites. However, while it is possible to
support forms, forms without JavaScript (using cookie
authentication) introduce the prospect of CSRF attacks. A good solution
for FastBoot forms that is also secure is probably a longer-term
project. We would encourage the community to experiment with a secure
approach to forms that works with FastBoot.

## jQuery Compatibility

In our RFC, we mentioned that dropping IE8 will give us the opportunity
to remove jQuery as a strict dependency. We should have been clearer
that we have no intent to remove the Ember APIs that delegate to jQuery
(such a Ember.$ and this.$() inside components).

Because these APIs will remain in 2.0, both for ease of upgrade and
because we have not yet made the jQuery dependency optional, Semantic
Versioning prohibits us from removing them until at least Ember 3.0.

On a personal note, we rely on jQuery heavily in our own apps. We think
it's a great library that remains hugely valuable to smooth over clunky
DOM APIs and browser quirks (even in modern browsers). For those users
who need the absolute smallest payload size, we don't want to saddle
you with a dependency that you don't need. But we expect the majority
of users to continue using jQuery, and we have no plans to remove the
Ember/jQuery integration at this time.

Thank you again for everyone who took the time to help us make this
decision, and thank you so much for being a part of the Ember
community.

[rfc-ie8]: https://github.com/emberjs/rfcs/pull/45

repository: SergeAstapov/ember-blog

gha_license_id: NOASSERTION

repo_name: SergeAstapov/ember-blog


text:
![OmniSci](https://raw.githubusercontent.com/jrajav/omnisci-rs/e2cc0b230871e098ff98768def5eb1429ba43a7b/omnisci.png)

-----

[![Latest Version](https://img.shields.io/crates/v/omnisci.svg)](https://crates.io/crates/omnisci)

A Rust client for connecting to [OmniSciDB](https://github.com/omnisci/omniscidb) via its RPC protocol of [Thrift](https://thrift.apache.org/). This package takes care of creating a client with the Thrift binary protocol and buffered transport, and exposes all of the OmniSci Thrift types and methods for use with it.

## Requirements

This client has been tested on these versions, and is expected to work with more recent versions unless otherwise noted:

* Rust 1.41+
* OmniSciDB 5.1+

It is also likely to work on earlier versions of OmniSciDB, but this is not officially supported.

## How to use

Add `omnisci` to your `Cargo.toml`:

```toml
[dependencies]
omnisci = "0.1.0"
```

Then create a client and connect (to the 'Backend TCP' port on the OmniSciDB instance):

```rust
use omnisci;

let mut client = omnisci::client::create("<IP_ADDRESS>:6274")?;

let user = "admin".to_string();
let passwd = "HyperInteractive".to_string();
let database = "omnisci".to_string();

let session = client.connect(user, passwd, database)?;

let query = "SELECT * FROM flights_donotmodify LIMIT 5".to_string();
let columnar = false;
let nonce = "1".to_string();
let first_n = 10000;
let at_most_n = -1;

let results = client.sql_execute(session, query, columnar, nonce, first_n, at_most_n);
```

See the examples folder for a more complete example.

## Contributing

### Building

This project is built with [Cargo](https://github.com/rust-lang/cargo), using `cargo build`. Each file in src/ except for lib.rs is automatically generated using Thrift, based on the the Thrift definitions stored at [omniscidb](https://github.com/omnisci/omniscidb) ([mapd.thrift](https://github.com/omnisci/omniscidb/blob/master/mapd.thrift), plus the .thrift files it references via include). To regenerate them, install Thrift 0.13.0 with your system's package manager, clone [omniscidb](https://github.com/omnisci/omniscidb) locally, and then run:

```
./generate_thrift_bindings.sh ../omniscidb
```

All source is also formatted with [rustfmt](https://github.com/rust-lang/rustfmt), via `cargo fmt`.

### Testing

The integration tests expect a local running instance of OmniSciDB on the default Backend TCP port: http://localhost:6274

If running natively (such as with the `./startomnisci` script), that port is accessible by default. If running with Docker, expose the Backend TCP port in addition to the default Frontend Web port using `-p 6274:6274`.

When ready, run `cargo test`.

## License

This project is licensed under the MIT license.

repository: mikehinchey/omnisci-rs

repo_name: mikehinchey/omnisci-rs


text:
feedback

========

feedback

Installation

------------

#####1. The preferred way to install this extension is through [composer](http://getcomposer.org/download/).

Either run

```

php composer.phar require --prefer-dist itshkacomua/yii2-feedback "*"

```

or add

```

"itshkacomua/yii2-feedback": "*"

```

to the require section of your `composer.json` file.

#####2. In the console of the root directory of the site, run the command
 ```
 php yii migrate --migrationPath=@itshkacomua/feedback/migrations/
```


text:
# Hello-World

## Authors
- [Jane Jepkemboi](https://github.com/jane95924)
- Add your clickable username here. It should point to your GitHub account.

## Project setup instructions
To start using this project use the following commands:

- `git clone https://github.com/jane95924/hello-world.git`
- `cd hello-world`
- `atom .`

## License info
MIT License

Copyright (c) 2020 jane jepkemboi

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.