text:
# Change Log

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html)

## [0.1.69] - 2023-06-07
we have added partitions for every batch we write to s3 to avoid file already existing errors. and added logging for run job flow request response.

Changed
api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
core/src/main/scala/core/DataFrameFromTo.scala

## [0.1.68] - 2023-06-07

Fixed bug of missing bracket in the datapulltask file.

### Changed
- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java

## [0.1.67] - 2023-06-18

Added functionally to configure hdfs-site.xml from the Input JSON.

### added

- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
- api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java

## [0.1.66] - 2023-06-07

Fixed bug of failureemailaddress not being passed in the correct argument order to performmigration
Moved send email call for failureemailaddress outside of min/maxexecutiontime check

### Changed

- core/src/main/scala/core/Controller.scala
- core/src/main/scala/core/DataPull.scala

## [0.1.65] - 2022-10-03

ClusterProperties can now create EMR cluster configuration fields in the "spark-defaults" classification.
ClusterProperties can now create EMR cluster configuration fields in the "spark-metrics" classification.

### Changed

- api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java

## [0.1.64] - 2022-09-21

ClusterProperties can now create EMR cluster configuration fields in the "spark-env" classification.

### Changed

- api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java

## [0.1.63] - 2022-07-23

Adding functionality for adding custom bootstrap file while spinning up the EMR cluster.

### Changed

- api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
- core/src/main/resources/Samples/Input_Json_Specification.json

## [0.1.62] - 2022-07-21

Adding examples for SFTP in documentation.

### Added

- core/src/main/resources/Samples/Input_Sample_s3_SFTP.json
- docs/docs/sftp.md

## [0.1.61] - 2022-07-18

Bumping emr release version for log4j vulnerability issue.

### Changed

- api/src/main/java/com/homeaway/datapullclient/config/EMRProperties.java

## [0.1.60] - 2022-06-24

Adding functionality for adding custom bootstrap file while spinning up the EMR cluster.

### Changed

- api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
- core/src/main/resources/Samples/Input_Json_Specification.json

## [0.1.59] - 2022-06-16

Adding functionality for sending custom arguments instead of default arguments through datapull API

### Changed

- api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
- core/src/main/resources/Samples/Input_Json_Specification.json

## [0.1.58] - 2022-06-16

Fixed `"logging_level"` in `"jdbcoptions"`

### Changed

- core/src/main/scala/helper/Helper.scala

## [0.1.57] - 2022-06-13

Update log4j version to avoid security vulnerabilities.

### Changed

- core/pom.xml
- functional-test/pom.xml

## [0.1.56] - 2022-04-29

This is to add a feature for adding spark session properties for a job.

### Changed

- core/src/main/resources/Samples/Input_Json_Specification.json
- core/src/main/scala/core/DataPull.scala
- core/src/main/scala/core/Migration.scala

## [0.1.55] - 2022-03-09

Adding Hive support for Core

### Added

- core/src/main/resources/Samples/Input_Sample_s3-to-hive.json

### Changed

- core/src/main/resources/Samples/Input_Json_Specification.json
- core/src/main/scala/core/DataFrameFromTo.scala
- core/src/main/scala/core/Migration.scala
- api/pom.xml

### Removed

- core/src/main/resources/Samples/Input_Sample_hive-to-filesystem

## [0.1.54] - 2022-03-03

Adding Hive as source platform

### Added

- core/src/main/resources/Samples/Input_Sample_hive_to_s3.json

### Changed

- core/src/main/resources/Samples/Input_Json_Specification.json
- core/src/main/scala/core/DataFrameFromTo.scala
- core/src/main/scala/core/Migration.scala

## [0.1.53] - 2022-02-19

Adding Hive support for API

### Changed

- api/src/main/java/com/homeaway/datapullclient/config/EMRProperties.java
- api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
- config yaml files

## [0.1.52] - 2022-02-18

This is the fix to a bug to make bootstrapactionstring work.

### Changed

- api/src/main/java/com/homeaway/datapullclient/process/DataPullRequestProcessor.java
- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
- api/src/main/java/com/homeaway/datapullclient/config/EMRProperties.java
- api/src/main/resources/application-dev.yml
- api/src/main/resources/application-prod.yml
- api/src/main/resources/application-stage.yml
- api/src/main/resources/application-test.yml
- master_application_config-dev.yml
- master_application_config-prod.yml
- master_application_config-stage.yml
- master_application_config-test.yml

## [0.1.51] - 2022-02-18

THis is add an option to chose logging level by the user for teradata datstores.

### Changed

- core/src/main/scala/helper/Helper.scala
- core/src/main/scala/core/DataFrameFromTo.scala
-  core/src/main/resources/Samples/Input_Json_Specification.json

## [0.1.50] - 2022-01-28

This is the fix to a bug in inline expression for jdbc code.

### Changed

- core/src/main/scala/helper/Helper.scala

## [0.1.49] - 2022-01-28

This is to add the functionality for inline expression evaluation for rdbms data stores.

### Changed

- core/src/main/scala/core/Migration.scala
- core/src/main/scala/core/DataFrameFromTo.scala
- core/src/main/scala/helper/Helper.scala

## [0.1.48] - 2021-12-15
This is to upgrade the log4j to fix vulnerability

Changed
- core/pom.xml

## [0.1.47] - 2021-11-17

This is to accept additional spark conf from the user for any migration

### Changed

- core/src/main/scala/core/Migration.scala
- core/src/main/scala/core/DataPull.scala

## [0.1.46] - 2021-11-17

This is to fix the bug which is throwing error while type casting the coalesce file count for fs writer.

### Changed

- core/src/main/scala/core/Migration.scala
- core/src/main/scala/core/DataFrameFromTo.scala

## [0.1.45] - 2021-11-16

Adding optional parameter for the user to configure subnet in which the EMR cluster can be spun up.

### Changed

-  api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
-  api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
-  core/src/main/resources/Samples/Input_Json_Specification.json

## [0.1.44] - 2021-09-27

-- fix cluster optional forcerestart

### Changed
-  api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java

## [0.1.43] - 2021-09-22

-- adding cluster optional forcerestart in datapull

### Changed
-  api/src/main/java/com/homeaway/datapullclient/config/EMRProperties.java
-  api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
-  api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
-  core/src/main/resources/Samples/Input_Json_Specification.json

## [0.1.42] - 2021-08-03

- Fixed a bug which was mandating aws secret access key for every migration

### Changed

- core/src/main/scala/core/DataPull.scala

## [0.1.41] - 2021-07-29

- Added optional attributes to make it easier to process Kafka streams created by MongoDB kafka connect source
- Added support for sql queries (jdbc and spark sql queries) to passed in as sql files from filesystem/S3
- Fixed bug in AWS Secrets Manager support, and added support for `inlinesecret{{}}`
- Increased timeout for API health check
- Fix bug of occasionally expiring CloudWatch log tokens
- Fix bug of EMR clusters not being re-used occasionally

### Changed

- Added optional attributes to make it easier to process Kafka streams created by MongoDB kafka connect source
  - core/src/main/scala/core/DataFrameFromTo.scala
  - core/src/main/resources/Samples/Input_Json_Specification.json
- Added support for sql queries (jdbc and spark sql queries) to passed in as sql files from filesystem/S3
  - core/src/main/scala/core/DataPull.scala
  - core/src/main/scala/helper/Helper.scala
  - core/src/main/resources/Samples/Input_Json_Specification.json
  - core/src/main/scala/core/Migration.scala
- Fixed bug in AWS Secrets Manager support, and added support for `inlinesecret{{}}`
  - core/src/main/scala/core/Controller.scala
  - core/src/main/scala/helper/Helper.scala
  - core/src/main/scala/core/Migration.scala
  - core/src/main/scala/security/SecretService.scala
  - core/src/main/scala/security/SecretsManager.scala
  - core/src/main/scala/security/SecretStore.scala
  - core/src/main/scala/security/Vault.scala
  - docs/docs/access_secrets.md
  - docs/mkdocs.yml
- Increased timeout for API health check
  - api/terraform/datapull_task/datapull_ecs.tf
- Fix bug of expiring CloudWatch log tokens
  - core/src/main/scala/logging/DataPullLog.scala
- Fix bug of EMR clusters not being re-used occasionally
  - api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java

## [0.1.40] - 2021-06-30

- Bug fix to pick up streaming parameters
- Sequencefile write with long key and default/snappy/deflate codecs

### Changed

- Bug fix to pick up streaming parameters
  - core/src/main/scala/core/Migration.scala
- Sequencefile write with long key and snappy codec
  - core/src/main/scala/core/DataFrameFromTo.scala
  - core/docker_spark_server/Dockerfile
  - core/src/main/resources/Samples/Input_Json_Specification.json

## [0.1.39] - 2021-06-24

- Bug fix to pick up streaming parameters
- Support for inline Spark SQL expressions in input json and sequencefile format
- Support for Structured Streaming watermarks (for stream-to-stream left joins, etc.)

### Changed

- Bug fix to pick up streaming parameters
  - core/src/main/scala/core/Migration.scala
- Support for inline Spark SQL expressions in input json and sequencefile format
  - core/src/main/scala/core/DataFrameFromTo.scala
  - core/src/main/scala/core/DataPull.scala
  - core/src/main/scala/helper/Helper.scala
  - manual-tests/filesystem_dataset_to_elasticsearch_to_filesystem/README.md
  - manual-tests/filesystem_dataset_to_elasticsearch_to_filesystem/datapull_input.json
  - core/src/main/resources/Samples/Input_Json_Specification.json
- Support for Structured Streaming watermarks (for stream-to-stream left joins, etc.)
  - core/src/main/scala/core/DataFrameFromTo.scala
  - core/src/main/resources/Samples/Input_Json_Specification.json

## [0.1.38] - 2021-06-14

- Update jettison dependency

### Changed

- core/pom.xml

## [0.1.37] - 2021-06-14

- Upgraded ABRiS library
- Support for S3 streaming destination

### Changed

- core/pom.xml
- core/src/main/scala/helper/Helper.scala
- core/src/main/scala/core/DataFrameFromTo.scala
- core/src/main/scala/core/Migration.scala

## [0.1.36] - 2021-06-08

- Re-added AWS bom for dependency management
- Fixed bug with SES config
- Added common options to Snowflake example
- Better error message when sql attribute in input json provided without query attribute
- Added required additional dependencies for jackson v 2.12.2
- Update Spark 2.4.7 local spark clusters to 2.4.8
- Updated documentation to fix embedded html issues
- Support for Kafka topics that aren't registered with Schema Registry

### Changed

- core/src/main/scala/config/AppConfig.scala
- core/pom.xml
- core/src/main/resources/Samples/Input_Sample_snowflake-to-filesystem-to-snowflake.json
- core/src/main/scala/core/Migration.scala
- api/pom.xml
- Update Spark 2.4.7 local spark clusters to 2.4.8
  - **/README.md
  - core/src/main/resources/Samples/Input_Json_Specification.json
  - core/Docker_Spark_Server/Dockerfile
- docs/docs/install_on_aws.md
- core/src/main/scala/core/DataFrameFromTo.scala

## [0.1.35] - 2021-05-25

- Made Consul datacenter Urls, AWS tags config-driven
- Added tags to ECR repo
- Upgraded API docker base image from non-supported version
- Removed extra leading slash for vault url prefix

### Changed

- Made Consul datacenter Urls, cost center tag config-driven
    - core/src/main/scala/config/AppConfig.scala
    - api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
    - core/src/main/scala/helper/Consul.scala
    - api/terraform/datapull_task/datapull_ecs.tf
    - api/src/main/java/com/homeaway/datapullclient/process/DataPullRequestProcessor.java
    - api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
    - api/terraform/datapull_task/ecs_deploy.sh
    - api/terraform/datapull_task/ecs_deploy_uninstall.sh
    - core/src/main/scala/core/Migration.scala
    - api/terraform/datapull_task/variables.tf
- Added tags to ECR repo
    - api/terraform/datapull_iam/datapull_user_and_roles.tf
- Upgraded API docker base image from non-supported version
    - api/Dockerfile
- Removed extra leading slash for vault url prefix
    - core/src/main/scala/security/Vault.scala

## [0.1.34] - 2021-05-18

Upgraded jackson and aws dependencies; standardised iswindowsauthenticated attribute

### Changed

- Upgraded jackson and aws dependencies
    - **/pom.xml
- standardised iswindowsauthenticated attribute
    - core/src/main/scala/core/DataPull.scala
    - core/src/main/scala/core/Migration.scala
    - core/src/main/resources/Samples/Input_Sample_teradata-to-filesystem.json

## [0.1.33] - 2021-05-18

Removed env var that clashes with fargate, fixed support for s3 service endpoints by adding support for fileprefix and schema for filesystem-like stores, updated manual test to cover explicitly specified schema

### Changed

- Remove env var that clashes with fargate
    - api/terraform/datapull_task/datapull_ecs.tf
- fixed support for s3 service endpoints by adding support for fileprefix and schema for filesystem-like stores
    - core/src/main/scala/core/DataFrameFromTo.scala
    - core/src/main/scala/logging/DataPullLog.scala
    - core/src/main/scala/core/Migration.scala
- updated manual test to cover explicitly specified schema
    - manual-tests/filesystem_stream_to_kafka/datapull_input.json
    - manual-tests/filesystem_stream_to_kafka/README.md

## [0.1.32] - 2021-05-15

- Upgraded Elasticsearch maven dependency to support ES 7.12.1, removed, unused dependencies
- Fixed https://github.com/homeaway/datapull/issues/81
- Added manual test for Elasticsearch sources and destinations
- Fixed Elasticsearch bugs, added support for additional optional ES Options and associated documentation
- Made Elasticsearch port, mapping, etc. optional with defaults

### Changed

- Upgraded Elasticsearch maven dependency to support ES 7.12.1, removed, unused dependencies
    - core/pom.xml
- Fixed https://github.com/homeaway/datapull/issues/81
    - core/src/main/resources/Samples/Input_Sample_*.json
    - functional-test/src/main/resources/dev_global-config.json
    - functional-test/src/main/resources/test_global-config.json
    - docs/docs/transformation.md
- Fixed Elasticsearch bugs, added support for additional optional ES Options and associated documentation
    - core/src/main/scala/core/DataFrameFromTo.scala
    - core/src/main/resources/Samples/Input_Json_Specification.json
- Made Elasticsearch port, mapping, etc. optional with defaults
    - core/src/main/scala/core/Migration.scala

### Added
- Added manual test for Elasticsearch sources and destinations
    - manual-tests/filesystem_dataset_to_elasticsearch_to_filesystem/*

## [0.1.31] - 2021-05-12

Remove bintray-bound Neo4j lib implementation until https://neo4j.com/product/connectors/apache-spark-connector/?ref=blog can be implemented

### Changed

- core/src/main/resources/Samples/Input_Json_Specification.json
- core/src/main/resources/Samples/Input_Sample_Oracle_PostPreMigration.json
- core/src/main/scala/core/DataFrameFromTo.scala
- core/src/main/scala/core/Migration.scala
- core/pom.xml
- docs/docs/index.md
- functional-test/src/main/resources/dev_global-config.json
- functional-test/src/main/resources/test_global-config.json
- functional-test/Jenkinsfile
- functional-test/pom.xml
- functional-test/testng.xml

### Deleted

- core/src/main/resources/Samples/Input_Sample_filesystem_to_Neo4j.json
- core/src/main/resources/Samples/Input_Sample_Neo4j.json
- core/src/main/scala/helper/Neo4j.scala
- functional-test/src/main/java/com/homeaway/utils/db/Neo4j.java
- functional-test/src/main/resources/input_files/MySQL-to-Neo4j.json
- functional-test/src/main/resources/input_files/Neo4jRelations.json
- functional-test/src/test/java/com/homeaway/DataPull/DataPullMySQLToNeo4JTest.java

## [0.1.30] - 2021-05-11

Fixing small bugs and making the brand, costcenter, application tags as optional; definition of dockerised spark server; installation instructions for Oracle and Teradata drivers

### Changed

- api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
- api/src/main/resources/input_json_schema.json
- core/docker_spark_server/Dockerfile
- docs/docs/oracle_teradata_support.md

## [0.1.29] - 2021-04-29
Added manual tests for mysql and mssql data stores, added support for secure Kafka schema registries, give ownership of files to bucket owners when writing to S3, reduced code redundancy, added permissions for bootstrap files and ECS tags, permission for default and custom EC2 EMR roles to send SES emails, support for custom AWS Service Endpoints like IBM Object Storage, made BucketOwnerFullControl default but removable

### Added
- manual-tests/mssql_dataset_to_mysql_to_mssql/* - Manual test for batch and pre/post-migrate commands on mysql and mssql data stores

### Changed
- DataFrameFromTo.scala - Added support for secure Kafka schema registries, give ownership of files to bucket owners when writing to S3, reduced code redundancy, made BucketOwnerFullControl default but removable
- Helper.scala - Added support for secure Kafka schema registries
- Input_Json_Specification.json - Added documentation for Teradata type (fastexport, fastload, etc), documentation for custom AWS Service Endpoints like IBM Object Storage, made BucketOwnerFullControl default but removable
- Migration.scala - Removed debug print that can cause credentials to leak to logs
- datapull_user_and_roles.tf - Added permissions for bootstrap files and ECS tags, and for sending SES emails
- Controller.scala - Set SES From email name to DataPull
- sample_custom_emr_ec2_role.tf - Added permissions for sending SES emails
- DataPull.scala - support for custom AWS Service Endpoints like IBM Object Storage, made BucketOwnerFullControl default but removable

## [0.1.28] - 2021-04-13
Converted Kafka source to use ABRiS and support spark streaming, support streaming filesystem source, add Console as a destination for batch and stream

### Added
- manual-tests/filesystem_stream_to_kafka/* - Manual test for streaming filesystems and Kafka sources

### Changed
- core/src/main/resources/Samples/Input_Json_Specification.json - Added documentation to support SSL Kafka sources and destinations,  plus update missing documentation for some filesystem actions
- Sample input json files - Updated to reflect change in sample CSV data file location
- core/src/main/scala/core/DataFrameFromTo.scala - Converted Kafka source to use ABRiS and support spark streaming, , support streaming filesystem source
- core/src/main/scala/core/Migration.scala - add Console as a destination for batch and stream, support streaming etc for file source and Kafka; refactored some function calls to use explicit parameter names
- core/src/main/scala/helper/Helper.scala - Added support for Kafka as a source, using ABRiS
- core/src/test/scala/DataFrameFromToFileSystem.scala - updated default value of mergeSchema option for parquet files, to false from null
- core/src/main/scala/core/DataPull.scala Removed unused code

## [0.1.27] - 2021-04-12
### Changed
- Controller.scala - Throw an exception to end the execution with exit code 1 when at least one migration has an error.

## [0.1.26] - 2021-04-12
### Changed
- api/src/main/java/com/homeaway/datapullclient/config/DataPullClientConfig.java
- api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
- api/src/main/java/com/homeaway/datapullclient/process/DataPullRequestProcessor.java
- api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java

## [0.1.25] - 2021-03-24
### changed
- Migration.scala - Fix port issue

## [0.1.24] - 2021-03-20
Added support for string event format for Kafka destinations
### Added
- .gitattributes - Force *.sh files to be handled with LF line endings regardless of OS, by git
### Changed
- core/src/main/scala/core/{DataFrameFromTo.scala, Migration.scala} - Added support for string event format for Kafka destinations
- core/src/main/resources/Samples/Input_Sample_s3_to_kafka.json - Updated example to use string format for kafka event key
- core/src/main/resources/Samples/Input_Json_Specification.json - Updated spec to support string event format for Kafka destinations
- manual-tests/filesystem_dataset_to_kafka/{README.md, datapull_input.json} - Updated manual test to test for string event format
- api/terraform/*/*.sh - changed line endings from CRLF to LF
- api/terraform/datapull_task/ecs_deploy.sh - removed `exit 0` from some previous debug accidentally committed

## [0.1.23] - 2021-03-14
### Added
- manual-tests/filesyste_dataset_to_kafka* - Added manual test instructions to test data movement from filesystem to kafka topic
### Changed
- core/pom.xml - Removed redundant dependency that causes runtime conflict for kafka clients
- manual-tests/README.md - Centralised steps to confirm datapull works locally, prior to doing the manual tests
- README.md, manual-tests/filesystem_dataset_with_uuids_to_mongodb/README.md - Updated spark submit command with necessary kafka runtime dependency

## [0.1.22] - 2021-03-08
### Added
- manual-tests/* - Added manual test instructions and supporting material

### Changed
- core/docker_spark_server/Dockerfile - Updated the download mirrors and versions for Apache Spark and Hadoop binaries
- pom.xml - Updated hadoop version to commonly supported version
- README.md - Updated with referece to manual tests

## [0.1.21] - 2021-02-25
### Changed
- core/src/main/scala/core/DataFramFromTo.scala - No longer using asInstanceOf[MongoClientURI]

## [0.1.20] - 2021-02-24
- core/src/main/scala/core/DataFramFromTo.scala - Fixed few arguments w.r.t the building kafka properties
- core/src/main/scala/core/Migration.scala
- core/src/main/scala/helper/Helper.scala

## [0.1.19] - 2021-02-12
### Changed
- api/src/main/java/com/homeaway/datapullclient/config/EMRProperties.java: Upgraded EMR version to 5.31.0
- core/docker_spark_server/Dockerfile: Upgraded Spark version to 2.4.6, downgraded Hadoop version to 2.10.0
- core/src/main/resources/Samples/Input_Json_Specification.json: Upgraded Kafka destination spec to match abris 4.0.1 capabilities
- core/src/main/resources/Samples/Input_Sample_Join_Heterogeneous_Sources.json: Fixed file spelling
- core/src/main/resources/Samples/Input_Sample_s3_to_kafka.json: Upgraded example to match abris 4.0.1 capabilities
- core/src/main/scala/core/DataFramFromTo.scala: Removed unnecessary println's, Use s3a when running locally, s3 otherwise, upgraded ABRiS to 4.0.1
- core/src/main/scala/core/DataPull.scala: Use s3a when running locally, s3 otherwise
- core/src/main/scala/core/Migration.scala: Removed unnecessary println's, upgraded ABRiS to 4.0.1
- pom.xml: Upgraded Spark version to 2.4.6, downgraded Hadoop version to 2.10.0, removed shaded jar, upgraded ABRiS to 4.0.1
- README.md: Upgraded Spark version to 2.4.6, downgraded Hadoop version to 2.10.0

## [0.2.1] - 2021-01-26
### Changed
api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java: S3 Cross-Account Access Support.

## [0.2.0] - 2020-12-22
### Changed
- Fixed install script error, upgraded AWS CLI to v2
- Fixed the MS DOS batch script for IAM principals
- Removed unnecessary environment variables
### Removed
- Removed the MS DOS batch script for ECS Deploy

## [0.1.18] - 2020-11-05
### Changed
- Fixed issue of Teradata destination deadlocking

## [0.1.17] - 2020-10-19
### Changed
- core/pom.xml : Remove hive JDBC dependency that was causing Spark SQL to fail
- core/src/main/scala/core/DataFrameFromTo.scala, core/src/main/scala/core/DataPull.scala, core/src/main/scala/core/Migration.scala, core/src/main/scala/logging/DataPullLog.scala, core/src/test/scala/DataFrameFromToFileSystem.scala : Added support RSA key based auth for SFTP
- core/src/main/resources/Samples/Input_Sample_filesystem-to-sftp.json : Updated to show RSA Key based auth
- core/src/main/resources/Samples/Input_Json_Specification.json : Added documentation for using sftp with password or RSA key auth

## [0.1.16] - 2020-10-18
### Changed
- README.md : Minor path fixes for running DataPull in a local Docker container
- core/pom.xml, core/src/main/scala/core/DataFrameFromTo.scala, core/src/main/scala/core/Migration.scala : Added support for Snowflake
- core/src/main/resources/Samples/Input_Sample_sftp-to-filesystem.json : Removed duplicate json attribute
- core/src/main/resources/Samples/Input_Json_Specification.json : Added documentation for uuing Snowflake as a source/destination
### Added
- core/src/main/resources/Samples/Input_Sample_snowflake-to-filesystem-to-snowflake.json : Added sample input json to read and write from Snowflake

## [0.1.15] - 2020-10-13
- Added support for reading/writing to a filesystem in orc.
- Added support for accepting an optional emr service role from the user if needed.
- Added support for accepting optional value and header fileds when writing data to kafka.
- Added support for reading/writing data to server side encrypted s3 buckets with AES-256.

### Changed
* api/src/main/java/com/homeaway/datapullclient/input/ClusterProperties.java
* api/src/main/java/com/homeaway/datapullclient/process/DataPullRequestProcessor.java
* api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
* api/terraform/datapull_task/ecs_deploy.sh
* core/pom.xml
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/DataPull.scala
* core/src/main/scala/core/Migration.scala

## [0.1.14] - 2020-09-23
- Added script to uninstall DataPull non-IAM components
- Reduced access of IAM Roles and Users used by DataPull
- Updated documentation for custom EMR profile role
- Added uninstall scripts and documentation for AWS deployments

### Added
* api/terraform/datapull_task/ecs_deploy_uninstall.sh: script to uninstall DataPull non-IAM components
* api/terraform/datapull_iam/uninstall_user_and_roles.sh: script to uninstall DataPull IAM components
* docs/docs/uninstall_on_aws.md: Documentation on how to uninstall DataPull in AWS

### Changed
* api/terraform/datapull_iam/datapull_user_and_roles.tfL Reduced access of IAM Roles and Users used by DataPull
* api/terraform/datapull_task/datapull_ecs.tf: Minro fix to spelling of AWS tags
* docs/docs/custom_emr_ec2_role.md: Updated documentation for custom EMR profile role
* docs/docs/install_on_aws.md: Updated documentation for custom EMR profile role
* docs/docs/index.md: Marked Kafka destination as supported datastore
* docs/mkdocs.yml: Updated documentation navigation

## [0.1.13] - 2020-09-16
- Added Kafka as a destination
- Fixed bug related to JDBC port not being read
- Added support for reading and writing S3 data when running in dockerised spark
- Minor improvements on time to install
- Cleaned up and updated dependencies

### Added
* core/docker_spark_server/Dockerfile: Run dockersied spark with Spark 2.4.4, Scala 2.11, Hadoop 3.x
* core/docker_spark_server/LICENSE: MIT License attribution to gettyimages for docker container
* core/src/main/resources/Samples/Input_Sample_filesystem-to-filesystem-avro.json: Sample for Avro file format

### Changed
* .gitignore: Ignore local log folder created when debugging
* README.md: Support for S3 sources and destinations when running as docker container
* api/src/main/java/com/homeaway/datapullclient/config/EMRProperties.java: Upgraded EMR version to 5.29
* api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java: Support for Kafka and Spark Avro
* api/terraform/datapull_iam/datapull_user_and_roles.tf: Additional permissions to EMR service role to support EMR 6.0.0
* api/terraform/datapull_task/ecs_deploy.sh: Installation speed improvement
* core/pom.xml: Cleaned up and upgraded dependencies; support for Kafka streaming
* core/src/main/resources/Samples/Input_Json_Specification.json: Documentation for Kafka as destination, EMR nodes
* core/src/main/resources/Samples/Input_Sample_s3_to_kafka.json: Updated sample for Kafka as destination
* core/src/main/scala/core/DataFrameFromTo.scala: Added support for Kafka; support to use S3 platform locally
* core/src/main/scala/core/Migration.scala: Added support for Kafka; bug fixes for JDBC port
* core/src/main/scala/helper/Neo4j.scala: Upgrade syntax to support Scala 2.12
* functional-test/pom.xml: Upgraded dependencies to resolve security advisories

### Deleted
* core/hive-jdbc-3.1.2.jar: Replaced downloaded dependency with maven reference

## [0.1.12] - 2020-08-05
### Changed
* core/pom.xml
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/Migration.scala

## [0.1.11] - 2020-07-28

### Added
* Added the jtds driver for supporting NTLM based windows authentication for sql server.
* Added a sample file [core/src/main/resources/Samples/Input_Sample_SQLServer_to_Cassandra_with_windows_authentication.json](Input_Sample_SQLServer_to_Cassandra_with_windows_authentication.json).
* Added the support for windows based based authentication for SQL Server.

### Changed
* Made necessary changes to the following files :
           * CHANGELOG.md
           * core/pom.xml
           *  core/src/main/resources/Samples/Input_Json_Specification.json
           *  core/src/main/scala/core/DataFrameFromTo.scala
           *  core/src/main/scala/core/Migration.scala

### Removed
* N/A

## [0.1.10] - 2020-07-07

### Added
* IAM Role emr_datapull_role and associated IAM Policies. This role is used as the default EMR service role, to fulfill Feature Issue #70

### Changed
* DataPull default install on AWS EMR and Fargate, spins up ephemeral EMR clusters of 6 m4.large instances instead of 3 m4.xlarge instances. This results in a spark clusters that cost the same but have 100% more usable CPU and memory
* DataPull default install on AWS EMR and Fargate, spins up ephemeral EMR clusters that use a EMR service IAM role emr_datapull_role with reduced permission needs , instead of using the default IAM Role EMR_DefaultRole that has very broad permissions. This fulfils Feature Issue #70
* Updated terraform for API's ALB listener with a commented line for the Certificate ARN; and fixed corresponding documentation that explains how to switch the API endpoint from HTTP to HTTPS
* DataPull default install on AWS EMR and Fargate, has the ECS Service with assign_public_ip flag as true instead of false; this is necessary for Fargate to fetch the ECR image (ref [AWS KnowledgeCenter](https://aws.amazon.com/premiumsupport/knowledge-center/ecs-pull-container-error/))
* Documentation to set up VPC etc in AWS Account for DataPull install fixes incorrect mention of <IP_ADDRESS>/24 as "2 IPv4 ranges"

### Removed
* Removed JDBC-based Hive data source functionality by reverting https://github.com/homeaway/datapull/commit/68478198b743aa4ebd26ed6151292fded7232b30 as a hotfix for Bug Issue #68
* In core/src/main/scala/logging/DataPullLog.scala, removed an unnecessary println

## [0.1.9] - 2020-07-06
### Changed
* api/terraform/datapull_task/ecs_deploy.sh - Fixed bug issue #67

## [0.1.8] - 2020-07-05
### Changed
* README.md - replaced instructions to install MkDocs with dockerised MkDocs commands; replaced need for JDK8 install for Dockerised execution,  with dockerised maven
* docs/docs/index.md - replaced instructions to install MkDocs with dockerised MkDocs commands

## [0.1.7] - 2020-06-11
### Added
* docs/docs/custom_emr_ec2_role.md
* docs/docs/resources/sample_custom_emr_ec2_role.tf

## [0.1.6] - 2020-05-22
### Changed
* core/src/main/resources/Samples/Input_Json_Specification.json
* core/src/main/scala/core/Controller.scala
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/DataPull.scala
* core/src/main/scala/core/Migration.scala
* core/src/main/scala/logging/DataPullLog.scala

## [0.1.5] - 2020-05-21
### Changed
* CHANGELOG.md
* core/pom.xml
* core/src/main/resources/Samples/Input_Json_Specification.json
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/Migration.scala

## [0.1.4] - 2020-05-15
### Changed
* core/pom.xml
* core/src/main/scala/core/Migration.scala

## [0.1.3] - 2020-05-04
### Changed
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/DataPull.scala
* core/src/main/scala/core/Migration.scala
* core/src/main/scala/logging/DataPullLog.scala

## [0.1.2] - 2020-05-04
### Changed
* core/src/main/scala/core/Migration.scala
* core/src/main/scala/core/DataFrameFromTo.scala

## [0.1.1] - 2020-05-04
### Changed
* core/src/main/scala/core/Migration.scala

## [0.1.0] - 2020-01-16
### Changed
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/Migration.scala
* core/src/main/scala/logging/DataPullLog.scala

## [0.0.9] - 2020-01-12
### Changed
* core/src/main/resources/Samples/Input_Json_Specification.json
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/Migration.scala

## [0.0.8] - 2020-01-10
### Added
* core/src/main/resources/Samples/Input_Sample_s3_to_kafka.json
### Changed
* core/src/main/resources/Samples/Input_Json_Specification.json
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/Migration.scala

## [0.0.7] - 2020-01-09
### Changed
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/Migration.scala

## [0.0.6] - 2020-01-03
### Added
* docs/docs/media/datapull_installation.png - added installation diagram

### Changed
* README.md - updated documentation on how to document
* core/pom.xml - updated version to fix a vulnerability
* docs/docs/index.md - moved logo
* docs/docs/install_on_aws.md - fixed formatting
* docs/docs/monitor_spark_ui.md - removed SSH tunnel diagram in documentation

## [0.0.5] - 2019-12-11
### Changed
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/core/Migration.scala
* core/src/main/scala/helper/Helper.scala

## [0.0.4] - 2019-12-10
### Added
* docs/docs/oracle_teradata_support.md
* docs/docs/aws_account_setup.md

### Changed
* api/pom.xml
* api/src/main/java/com/homeaway/datapullclient/config/DataPullClientConfig.java
* api/src/main/java/com/homeaway/datapullclient/config/DataPullClientConfig.java
* api/src/main/java/com/homeaway/datapullclient/config/DataPullProperties.java
* api/src/main/java/com/homeaway/datapullclient/config/EMRProperties.java
* api/src/main/java/com/homeaway/datapullclient/process/DataPullRequestProcessor.java
* api/src/main/java/com/homeaway/datapullclient/process/DataPullTask.java
* api/src/main/java/com/homeaway/datapullclient/start/DatapullclientApplication.java
* api/src/main/resources/application-dev/test/stage/prod.yml
* api/src/main/resources/overwrite_config.sc
* api/src/main/resources/read_application_config.sc
* api/src/main/resources/terraform/datapull_task/datapull_ecs.tf → api/terraform/datapull_task/datapull_ecs.tf
* api/terraform/datapull_task/ecs_deploy.sh
* api/terraform/datapull_task/variables.tf
* core/pom.xml
* core/src/main/resources/application-dev/test/stage/prod.yml
* core/src/main/scala/config/AppConfig.scala
* core/src/main/scala/core/DataFrameFromTo.scala
* core/src/main/scala/logging/DataPullLog.scala
* core/src/main/scala/security/Vault.scala
* docs/docs/install_on_aws.md
* master_application_config-dev/test/stage/prod.yml
* .gitignore

### Deleted
* api/src/main/resources/application.yml
* master_application_config.yml

## [0.0.3] - 2019-11-15
### Added
-Added Documentation - "Added the documentation for Monitoring spark UI for the datapull jobs."

## [0.0.2] - 2019-08-20
### Fixed
- Fixed bug - "create_user_and_roles.sh script is not generating access and secret keys"

## [0.0.1] - 2019-06-26
### Fixed
- Fixed "No json input available to Data Pull" exception when running DataPull from EMR
- Fixed create_user_and_roles.sh script

repository: homeaway/datapull

gha_language: Java

gha_license_id: NOASSERTION

repo_name: homeaway/datapull


text:
# System and user data backups and recovery
## Backups

How to backup btrfs subvolumes explained in a nice [Fedora Magazine article](https://fedoramagazine.org/btrfs-snapshots-backup-incremental/).
Here directory `/.snapshots` will be used for snapshots, and subvolume `backup`, mounted at `/backup` for copies of snapshots.
Better to send the snapshots to the different physical location, but here backup strategies not covered.
Snapshot dont include nested snapshots or subvolumes, it creates only empty directory.
So to perform snapshot of subvolume `/var/log`, must run command `btrfs sub create root/var/log /.snapshots`

Edit file  `/etc/fstab` by copying any included subvolume line, and edit it like example:
```properties
UUID=some_random_symbols   /backup  btrfs defaults,noatime,commit=30,space_cache,compress=zstd:4,subvol=backup 0 2
```
Create snapshot / backup of the root subvolume:
```bash
mkdir /backup
cd /run/btrfs-root
mkdir snapshots
btrfs sub create backup
mount /backup
btrfs sub snap -r root/ /run/btrfs-root/snapshots/root_fresh
# This is example hot to send snapshot to different location. It is possible to send read-only snapshots to the remote btrfs storage.
btrfs send /run/btrfs-root/snapshots/root_fresh/ | btrfs receive /backup/
```
This will not include other subvolumes like /home /opt, var/log and so on. Must snapshot (optimal: send to other physical disk) all important subvolumes.

Now we can sleep better without stress - if somethong goes wrong, will be posibble revert to last good snapshoped state.

## Recovery
Boot from live cd, decrypt storage:
Below is the example how to revocer root subvolume:
```bash
if [[ $(lsblk |  grep -vE "NAME|tmpfs|cdrom|loop|mapper" | awk '{print $1}' | head -n 1) != sda  ]]; then
  export DISK="/dev/nvme0n1"
else
  export DISK="/dev/sda"
fi
  DISKP="${DISK}$( if [[ "$DISK" =~ "nvme" ]]; then echo "p"; fi )"
DM="${DISK##*/}"
cryptsetup luksOpen ${DISK}3 ${DM}3_crypt
mount /dev/mapper/${DM}3_crypt /mnt
cd /mnt
btrfs sub del root
# Choose one variant from two:
# First - recover from the (remote) location:
btrfs send snapshots/root_fresh/ | btrfs receive .
mv root_fresh root
# Second - snapshot subvolume from local file system (restore very quick):
btrfs sub snap snapshots/root_fresh root
reboot
```
Only / (root) must be recovered from live cd. Other subvolumes recovering simple:
- stop services using subvolume;
- delete subvolume;
- snapshot from the napshot / backup;
- restart stoped services.
Example for postgresql subvolume:
```bash
ps -ef | grep postgresql
service stop postgresql
cd /run/btrfs-root
btrfs sub del postgresql
btrfs sub snap snapshots/postgresql postgresql
service start postgresql
```

license: MIT

branch_name: refs/heads/main

extension: md

license_type: permissive


text:
---
layout: post
title: FOSSASIA'16 Experience
tags: fossasia
subclass: 'post tag-gsoc'
navigation: True
logo: 'assets/images/ghost.png'
cover: 'assets/images/fossasia.jpg'
---

This year I  was invited to FOSSASIA'16 Singapore to give a talk on my GSoC'15 project I did for Python Software Foundation.

I attended the full conference (17th March - 20th March) and had a great experience while delivering my talk.

FOSSASIA is said to be ASIA's biggest opensource conference and it attracted many great speakers from across the globe. Thus it gave me an opportunity to meet those people in person whom I only knew by their IRC nicknames and blogs.

The best part of this conference, apart from the talks, was to have a one to one discussions with other speakers. People were really humble there. You could just walk to anyone, say Hi! and get started on a healthy discussion.

Stephanie Taylor, Cat Allmen from Google, who manages Google Summer Of Code programme, and I had a discussion on how we can further encourage younger students to get involved in open source.

I had a long conversation with [Mike McQuaid](https://github.com/mikemcquaid) (Maintainer of Homebrew). Talked about our open source projects and I took a lot of career advices from him.

Further I met a lot of other open source developers. Meeting and talking to these people motivated me a lot. These people don't code for making money, but for their passion and solving real life difficult problems with it.

For example, I met a school teacher who is teaching Physics in an interactive way using Python, met a guy who is making a code editor for blind people.. etc. There was a lot of energy in that conference, and it was inspiring to see people moving to open source to solve these problems.

My talk was on 19th March at 3 PM at *Pauli's Lab*. The crowd turnout was very impressive and I had a great time presenting my talk.

@yask123 enjoyed your talk mate
— Johanan Ottensooser (@oatsandsugar) March 19, 2016

Overall it was a wonderfull experience. Everything was well executed and I hope to be there next year too @ FOSSASIA'17.

Also, I would like to thank *Google* and *Python Software Foundation* for supporting me financially without which it wouldn't have been possible for me to attend this amazing event.

branch_name: refs/heads/master


text:
---
layout:     post
title:      "Gettin wild in Idyllwild"
subtitle:   "A day in paradise and a night of fun"
date:       "2017-05-19 21:04:28"
author:     "Randall"
header-img: "img/Gettin-wild-in-IdyllwildHeader.JPG"
mile:       "170"
---

After my first solo camping, I headed out into the hot dusty desert. It got warm fast and I was low on water. I had four miles to go before the first water source and made the mistake of eating some beef jerky in the morning. I ran out of water on the side trail to the spring. I filtered a couple liters of water for the rest of my day and kept on hiking. The water tasted like iron.

Later, in the heat of the day, I came upon a water cache and was excited to replace my water with something that tasted better. The cache was empty.

![photo0](/img/Gettin wild in IdyllwildPost0.JPG)

I kept on hiking when I found Walden. Walden is a little oasis managed by a local trail angel that celebrates Thoreau and Whitman. It even has a Little Free Library, where you can trade books. I sat there and snacked while I dried my tent from the previous nights condensation.

That's when Shoe showed up. He was desperate for water and said there was supposed to be water here. I hadn't seen any, but after some investigation we found a large cistern of cold water. I chugged my gross water and replaced it with the new good water.

That was enough to get me to my final destination, Paradise Valley Cafe. They're hiker friendly and will even let you camp out on their porch. When I arrived at the Cafe, I was the only hiker there. The waitress actually said I looked too clean to be a hiker.  I ordered a burger and a coke and enjoyed my cell phone service.

![photo1](/img/Gettin wild in IdyllwildPost1.JPG)

Then everyone showed up. Almost everyone I had talked to in the last few days showed up all at once. I had planned on sleeping there, but a few of us decided to hike the mile back to the trail and put in a few miles at night. The weather was perfect for hiking and it was cool watching the sun set in the desert.

![photo2](/img/Gettin wild in IdyllwildPost2.JPG)

The next section of trail goes on for about 15 miles before it closes due to a forest fire a few years ago. Lots of people skipped from the cafe straight to Idyllwild, but we hiked to the closure. From the closure, we took an old dirt road back to the highway where we hitchhiked into town.

![photo3](/img/Gettin wild in IdyllwildPost3.JPG)

First thing we did was go get pizza. We had met a girl on the trail a few days earlier who was obsessed with getting pizza in Julian, and that is how we felt all day.

After finishing our pizzas, we bought our food for the next few days and then took a much needed shower. I walked in with my clothes on and soaped them up. A simple way of doing the laundry.

We met up with most of the people from the day before at the cafe and hung out at the campground in town. Tomorrow, we summit Mt San Jacinto.

extension: markdown


text:
<% seoDescription("Contribute to Buffalo") %>
<% seoKeywords(["buffalo", "go", "golang", "contribute", "docs", "funding", "community", "framework", "web"]) %>

<%= h1("Contributing") %>

The Buffalo team needs, and wants your help! We're doing our best to make the world a better place for Go web development, but we can't do it alone.

If you want to contribute, please read this article first: [Contributing to Open Source Git Repositories in Go](https://splice.com/blog/contributing-open-source-git-repositories-go/). It shows how to configure your `git` environment to avoid common pitfalls. This article is recommended to all those who are looking to contribute to any Go projects.

## The Buffalo Project

[https://github.com/gobuffalo/buffalo](https://github.com/gobuffalo/buffalo)

##### Guidelines for Contributing

* When contributing to Buffalo please keep in mind that tests are **100%** required! No PR will be accepted without accompanying tests.
* CodeClimate is run against all PRs. We recommend you run it locally to save you time and hassle with the PR. [https://github.com/codeclimate/codeclimate](https://github.com/codeclimate/codeclimate)

## The Buffalo Docs

Contributing to this documentation site is a great, and easy, way to help make Buffalo better.

[https://github.com/gobuffalo/docs](https://github.com/gobuffalo/docs)

## Funding

If you like, or love, Buffalo, please consider funding its development.

[https://patreon.com/buffalo](https://patreon.com/buffalo)

With funding I will be able to focus on Buffalo development full-time. This means faster turn around times on bugs, issues, and features. It also means more time spend on documentation and learning materials such as blog posts, videos, and possibly a book.

## Shoulders of Giants

We are fortunate, as a community, to have so many great packages written by some really great people. Buffalo wraps all of those packages together with a little glue to bring it all together. A great place to start making Buffalo better is to help improve the underlying packages Buffalo relies on.

[https://github.com/gobuffalo/buffalo/blob/master/SHOULDERS.md](https://github.com/gobuffalo/buffalo/blob/master/SHOULDERS.md)

## Other Buffalo Projects

There are also many other official and non-official Buffalo projects to which you can contribute. Search for the [gobuffalo](https://github.com/topics/gobuffalo) topic on GitHub to discover new projects.

repository: gobuffalo/docs

gha_language: JavaScript

gha_license_id: MIT

repo_name: gobuffalo/docs


text:
## Install
* **Note: recommended a node version >= 10**

## Run

Download dependencies and start the app in the `local` environment:

```bash
$ yarn
$ yarn local
```

repository: mssodhi/node-boilerplate

repo_name: mssodhi/node-boilerplate


text:
## NOWPAP Eutrophication Assessment Tool (NEAT) Map Generator

This Python-based software is dedicated to generating the NEAT map through the use of WIMSOFT.
This tool is just an integrator of the many command line (CLI) WAM tools

### Setup of Python Environment
---
The only Python setup required is just the installation of the Python itself and a few libraries
used for number crunching and image printing.

#### Prerequisites
##### Required Modules
The software have been tested with the following module versions

- netCDF4==1.5.5
- numpy==1.20.0rc1
- matplotlib==3.1.3
- python-dateutil==2.8.1
- pyhdf==0.10.2
- coloredlogs==14.0
Aside Python modules, [WIMSOFT](https://wimsoft.com/) needs to be installed in order to have the scripts working

##### Running the main
- Once all is set, inside the ```main.py``` change the following parameters according to your environment.
  - fp - input file pattern that includes the full path
  - op - output path
  - pp - path where to output the png images generated
  - if - input files, those consistent of the trend file from wam_trend, the composite file, the count, etc.
  - dv - data variable. This is the name of the variable inside "if" above
  - by - the base year, i.e., the year in which the trend estimates start

- The above parameter stores sensor-based input parameters
For any function call within this tool, the input parameters or the user case can be infered through command line
  interface. All scripts are CLI and have the right documentation within
## Authors

Eligio Maure<EMAIL_ADDRESS>
## License
[MIT](https://opensource.org/licenses/MIT)

repository: npec/PyWAM

repo_name: npec/PyWAM


text:
# Apicco CRUD example

This is a sample app that you can create, update, delete and list movies.

You can try it as follows:

1. Start the server with `npm start`

2. Execute demo CRUD actions with `npm run demo`

repository: SokratisVidros/apicco

repo_name: SokratisVidros/apicco


text:
# city_explorer_api

# Lab 6

Number and name of feature: _____Lab: 06 - Node, npm, and Express___________

Estimate of time needed to complete: __4 hours__

Start time: __11:00___

Finish time: ___20:00__

Actual time needed to complete: __9 hrs___

**Author**: Ruben Arthur Lozano
**Version**: 1.0.0 (increment the patch/fix version number if you make more commits past your first submission)

## Overview
- This app takes in user input as far as a city and outputs the location coordinates and weather.

## Getting Started
- Know what the frontend dev needs
- Obtain proper Apis with weather and geo data
- Install node.js
- Build a server.js to connect the dots

## Architecture
<!-- Provide a detailed description of the application design. What technologies (languages, libraries, etc) you're using, and any other relevant design information. -->
- NPM
- Node.js
- VSCode

## Change Log
<!-- Use this area to document the iterative changes made to your application as each feature is successfully implemented. Use time stamps. Here's an examples:

01-01-2001 4:59pm - Application now has a fully-functional express server, with a GET route for the location resource.

## Credits and Collaborations
<!-- Give credit (and a link) to other people or resources that helped you build this application. -->
- 12-28-2020 20:09 Application has a fully-functional express server with several GET routes that access different data

# city_explorer_api

# Lab 7

Number and name of feature: _____Lab: 06 - Node, npm, and Express___________

Estimate of time needed to complete: __4 hours__

Start time: __11:00___

Finish time: ___20:00__

Actual time needed to complete: __9 hrs___

**Author**: Ruben Arthur Lozano
**Version**: 1.0.0 (increment the patch/fix version number if you make more commits past your first submission)

## Overview
- This app takes in user input as far as a city and outputs the location coordinates and weather.

## Getting Started
- Know what the frontend dev needs
- Obtain proper Apis with weather and geo data
- Install node.js
- Build a server.js to connect the dots

## Architecture
<!-- Provide a detailed description of the application design. What technologies (languages, libraries, etc) you're using, and any other relevant design information. -->
- NPM
- Node.js
- VSCode

## Change Log
<!-- Use this area to document the iterative changes made to your application as each feature is successfully implemented. Use time stamps. Here's an examples:

01-04-2021 11:59pm - Application now has a fully-functional express server, with a GET route for the location resource.

## Credits and Collaborations
<!-- Give credit (and a link) to other people or resources that helped you build this application. -->

# Lab 8
Number and name of feature: _____Lab: 07 - APIs___________

Estimate of time needed to complete: __4 hours__

Start time: __12:00___

Finish time: ___17:00__

Actual time needed to complete: __5 hrs___

**Author**: Ruben Arthur Lozano
**Version**: 1.0.0 (increment the patch/fix version number if you make more commits past your first submission)

## Overview
- This app takes in user input as far as a city and outputs the location coordinates and weather.

## Getting Started
- Know what the frontend dev needs
- Obtain proper Apis with weather and geo data
- Install node.js
- Build a server.js to connect the dots

## Architecture
<!-- Provide a detailed description of the application design. What technologies (languages, libraries, etc) you're using, and any other relevant design information. -->
- NPM
- Node.js
- VSCode

## Change Log
<!-- Use this area to document the iterative changes made to your application as each feature is successfully implemented. Use time stamps. Here's an examples:

01-7-2021 11:00pm - Application now has a fully-functional express server, with a GET route for the location resource.

## Credits and Collaborations
<!-- Give credit (and a link) to other people or resources that helped you build this application. -->
- 01-06-2021 21:09 Application has a fully-functional express server with several GET routes that access different data

# Lab 9

**Author**: Arthur Lozano
**Version**: 1.0.0 (increment the patch/fix version number if you make more commits past your first submission)

## Overview
Requests movies related to the area, yelp info, and events near the location.

## Getting Started
- Create routes
- get API information
- connect variables to keys within env
- Create handlers
## Architecture
- Express
- Heroku
- Superagent

## Change Log
<!-- Use this area to document the iterative changes made to your application as each feature is successfully implemented. Use time stamps. Here's an examples:

01-13-2021 11:30pm - Application now has a fully-functional express server, with a GET routs for yelp and a movie API.

## Credits and Collaborations
Davion Garcia, Nico, some TAs-->


text:
# On Track Detectors
Sensors to detect Rolling Stock on tracks

## Locomotive Detectors (DCC)
Current compsution Circuits to detect the presence of locomotives on track, also can detect wagons/cars that have some kind of current ussage (leds, resistors, accesory decoder,...)

* __[SMD Analogic and Digital](Locos/Detector_SMD_AD)__: Sensor with Digital Ouput and Analogic proportional to the current ussage.
* __[SMD Simple](Locos/Detector_SMD_Simple)__: Minimum Detector with digital output.

## Documents
On the [Documents](Doc) folder are some schematics found public on the internet, cited and used on the creation of this circuits, they have their own author, check them if you need.

## Simulation
Inside [Simulation](Simulation) there are Kicad Schematics for use with the simulation tool (ngspice). Used to understand the circuits and create the boards.

repository: danielvilas/DigitalTrains

repo_name: danielvilas/DigitalTrains


text:
# Exporting JSON Schema

One can export the JSON Schema to validate the mapped types.  `#include <daw/json/daw_json_schema.h>` and
call `daw::json::to_json_schema<MyType>( "identifier", "title" );` to generate a `std::string`. There is an output
iterator version too.
Refer
to [daw_json_schema_test.cpp](../../tests/src/daw_json_schema_test.cpp)
for a working example.

branch_name: refs/heads/release


text:
# PN5_60961

<PageHeader />

## Description

Add milestones to jlogdup.

### Previous Release Behavior

Due to unidirectional nature of jlogdup, it is messy to accurately know, from the sender's side, the progress (timestamp) of the receiving side, when jlogdup needs to be restarted.

### Current Release Behavior

Using reverse channel of socket connection, receiving side periodically sends the timestamp of the last record processed to the sender. Sender records this progress in a milestone file, which can be used to set the start time upon restart.

Back to [5.7.6 Release Notes](../jbase-5.7.6-release-notes/README.md)

<PageFooter />

repository: josepheshaw/docs

repo_name: josepheshaw/docs


text:
ViCorrection: Vietnamese Spelling Correction

## Overview
A project to correct spelling errors in Vietnamese texts using Seq2Seq and Seq2Seq-Attention models at character-level

More information can read from [training.ipynb](notebooks/Spelling_Correction_Vietnamese_Training.ipynb) or [testing.ipynb](notebooks/Spelling_Correct_Vietnamese_Testing.ipynb)

### Setup
```bash
pip install -r requirements.txt
```

### Create dataset
Change .csv path and save_path in dataset/create_dataset.py and run:
```bash
cd dataset
python create_dataset.py
```

### Model (character-level)
##### List of neural models in the repo:

- [```Seq2Seq```](weights/seq2seq_without_att.pth)
- [```Seq2Seq-BahdanauAttention```](weights/seq2seq.pth)

### Training
Edit parameters in params.py file and training:
```bash
python training.py
```

# Performances
| Spell
Checker    | CER (%) | Full Sequence Acc (%) |
|----------|----------------------|--------------------------------------|
| ```Seq2Seq``` | 1.34 | 82.3 |
|``` Seq2Seq-Attention``` | 1.12 | 85.7|


text:
# pssh-tools-cli

[![Build Status](https://travis-ci.org/feedsbrain/pssh-tools-cli.svg?branch=master)](https://travis-ci.org/feedsbrain/pssh-tools-cli)

**Command Line Tools to decode PSSH Data and PSSH Box**

For dealing with multi-drm using common encryption (cenc) we may need to encode or decode pssh data and/or pssh box to use in our workflow. This tools is written to help that process.

## Installation

This command line is installed via npm:

``` bash
$ npm install -g pssh-tools-cli
```

# CLI Help

``` bash
$ psshtools -h
```

## Supported PSSH

Currently we're only focus on Widevine and PlayReady but we will support more in the future.

repository: feedsbrain/pssh-tools-cli

repo_name: feedsbrain/pssh-tools-cli


text:
# timeron

big ol' timer that scales to whatever size you happen to resize it to. good if you have an awkward need for some huge digits.

counts down from whatever number you punch in to zero, then back up to 99:59 if you leave it alone long enough.

colors change between modes to help indicate current state even from far away!
* green: input. 0-9 adds time (think microwaves), backspace clears the last digit
* gray: ready to start
* blue: counting down
* red: countdown expired, counting up now

helpfully beeps when timer starts and when timer reaches 0:00

written in lua using [love2d](https://love2d.org), last tested in version 11.1. may not work in newer versions.

inputting time
![screenshot of inputting time into timer](http://i.imgur.com/4M1bk6f.png)

counting down
![screenshot of time counting down. somehow.](http://i.imgur.com/x3E9yHu.png)

time has expired and is now counting up
![screenshot of timer counting up after time expires](http://i.imgur.com/Ialigpo.png)


text:
# twitter-stalky

Also Hosted at www.twitter-stalky.herokuapp.com

A simple portal built on Ruby on Rails that fetches all the twitter posts for a given Twitter Handle.
Let the Stalking Begin!

Note: Twitter API allows only a maximum of 200 timeline fetches in one call and a maximum of 3200 posts fetches

#How to configure?

1. Please create an application at apps.twitter.com
2. Generate the Consumer Key(API Key) | Consumer Secret(API Secret) | Access Token| Access Token Secret
3. Mention the credentials obtained above in /config/initializers/twitter.rb
4. You are now good to go! Happy Stalking!

#TO-DO

1. Translation support using Google Cloud Translate API
2. Sentiment Analysis using HavenonDemand

#What goes  inside?

1. Twitter gem to fetch the posts
2. Simple_form gem for forms
3. Twitter BootStrap for UI
4. iFrames for Google Maps in footer
5. Font Awesome for Glyphicons


text:
Harker Schoology Chrome Extension

    A Chrome extension that adds awesome features to Schoology, made specifically for Harker students.

    By Aarav Borthakur and Aarush Vailaya

***

* [Features](#features)
* [Installation](#installation)
* [Usage](#usage)
* [Bugs and feature requests](#bugs-and-feature-requests)
* [Releases](#releases)
* [Contributing](#contributing)

***

## Features

-   ### Infinite campus schedule can now be found on Schoology.

    ![](images/schedule.gif)

-   ### Teachers' zoom links can now be found on Schoology (constantly updating).

    ![](images/zoom.gif)

-   ### "Overdue" section added to classes' pages.

    ![](images/overdue.png)

-   ### Weekly lunches can be found on Schoology.

    ![](images/lunch.gif)

-   ### Enhanced Schoology PDF viewer.

    ![](images/pdf.png)

-   ### Much more!

***

## Installation

Click [here](https://schoology-extension.herokuapp.com) to download the extension files (can take up to ten seconds to wake up app). Unzip the download.

Go to [chrome://extensions/](chrome://extensions/) and enable developer mode from to top right of your screen, to the right of the search bar.
![](images/developer-mode.png)

Drag the unzipped folder to the page, and that's it! You have the extension installed!

***

## Usage

To view your schedule and zoom links, hit your profile picture on the top right of any Schoology page. You must be signed in with your Harker email for this to work.
Hit "schedule" or "zoom links" in the dropdown menu. On your first time on your schedule page, you will be asked to login to Schoology again.

![](images/dropdown.png)

To see overdue assignments, enter a course's page and look to the right of your screen. The overdue assignments should appear above the "upcoming".

To see lunch, look to the top of any Schoology page, next to the search button. Hit the lunch button, and you will be directed to the week's lunch.

![](images/lunch-icon.png)

***

## Bugs and Feature Requests

Please email [good.aarav<EMAIL_ADDRESS>for bug reports and feature requests.

***

## Releases

You can view and install previous releases [here](https://github.com/Harker-Hackers/schoology-extension/releases).

***

## Contributing

This utility is completely open source! The source code can be found [here](https://github.com/Harker-Hackers/schoology-extension). Feel free to [contribute](https://gist.github.com/MarcDiethelm/7303312)!


text:
# PN5_60636

<PageHeader />

## Description

Record locking not case insensitive

### Previous Release Behavior

Under D3 emulation, item locks were not case insensitive.

READU/READVU/MATREADU would not respect locks on differently cased IDs.

### Current Release Behavior

Record locks are case insensitive.

We always store an upper case version of the ID.

Note: This case insensitivity matches D3 which means it is ALWAYS case insensitive regardless of CASING ON/OFF or CASE ON/OFF.

Back to [5.6.3 Release Notes](./../README.md)

<PageFooter />

repository: patpog/docs

repo_name: patpog/docs


text:
---
name: Documentation
about: Report problems with the documentation
title: "[Docs]"
labels: ''
assignees: ''

---

One of:

**Insufficient**: Something is documented, but the current documentation is inadequate

**Missing**: Something is undocumented. Note that most internal functions should be considered self-documenting -- consider submitting an enhancement report for refactoring if they are not.

**Other**: Something else. We will update this template to include your problem category afterwards.


text:
---
layout: archive
title: Pycon 2014 - Call for posters
excerpt: PyCon Call for contributions
year: 2013
month: 8
day: 7
published: true
---

PyCon is the premier Python Conference, and in 2014 it is being held for the first very time outside US - in the beautify Montéal/CA. It is not only a great opportunity to met the best python programmers across the world, but also to make the PyCon even better.

How?

What about presenting a poster in the Python premier conference?

# How does it work? #

As a presenter, you get to interact directly with your audience, share your passion and your idea, and immediately address questions in a more conversational manner. Moreover, the environment is generally less stressful and more easygoing than giving a formal talk. On top of that, you met a lot of interesting people.

The idea is to present your topic on a 4'x4' poster board and as attendees mingle through the rows, they find your topic, read through what you have written, the strike up a discussion on it.

Users of all levels go to the poster session, so your topic can focus on novice, intermediate or even expert programmers. Focus on what you think is interesting, rather than difficult/complex. This video will give you an idea of what poster sessions are.

# What kind of proposal can I submit? #

You may be wondering about what kind of python stuff you will be able to talk. As an advice, we can told you that you can talk about anything you want to. Don't be shy. We would love hear what make you exciting with Python. But, if the problem is missing ideas, why don't take a look in the poster section of [PyCon'13](https://us.pycon.org/2013/schedule/posters/list/) or [PyCon'12](https://us.pycon.org/2012/schedule/lists/posters/)? People made an incredible work in there.

# Hmm.. I'm interested. How to submit a proposal?

To submit a poster proposal, you should first create an account in the PyCon site. When you are logged in, you will need to include the following information.

- The title of your post

- The category

- The intended audience (assumed level of knowledge)

- A brief description - 300 chars max

- A detailed description

- Additional notes for the proposal reviewers. This can include additional background information such as links to other presentations you have done, or links to blog posts, articles or code related to your presentation.

Keep in mind that the posters should follow all of the usual [guidelines](http://us.pycon.org/2014/speaking/proposal_advice/) that we look for in talk and tutorial proposals.

# Important Dates

Do not miss the application deadline!

Poster submission deadline: November 1st, 2013.

Poster announced: December 15th, 2013.

Poster session at PyCon'14: April 13th, 2014.

If you need more information, you can see the [Poster FAQ](http://us.pycon.org/2014/posters/faq/) and the [Poster CFP](https://us.pycon.org/2014/posters/).

We look forward to seeing you in Montreal in 2014!


text:
<img src="https://cdn.rawgit.com/crisward/riot-grid/master/grid-logo.svg" height="140"/>

Checkout the new [RiotGrid2](https://github.com/crisward/riot-grid2)

[![Build Status](https://travis-ci.org/crisward/riot-grid.svg)](https://travis-ci.org/crisward/riot-grid)
[![Coverage Status](https://coveralls.io/repos/crisward/riot-grid/badge.svg?branch=master&service=github&v=3)](https://coveralls.io/github/crisward/riot-grid?branch=master)
[![NPM Downloads](https://img.shields.io/npm/dm/riot-grid.svg)](https://www.npmjs.com/package/riot-grid)
[![NPM Downloads](https://img.shields.io/npm/v/riot-grid.svg)](https://www.npmjs.com/package/riot-grid)

## Installing

```
npm install riot-grid
```

## Usage

```html
<grid tabindex="1" data="{collection}" height="{400}" click="{handleSelect}" dblclick="{handleDoubleClick}" onchange="{handleSelectionChange}">
  <gridhead>
    <span style="width:70%">Name</span>
    <span style="width:30%">Age</span>
  </gridhead>

  <gridbody>
    <span style="width:70%">{row.name}</span>
    <span style="width:30%">{row.age}</span>
  </gridbody>

</grid>

```

## Demo

See demo on [codepen](http://codepen.io/crisward/pen/rxepMX?editors=101)
This includes a bit of code to generate 100,000 random users to show off the performance of the grid.
I tried increasing this to 10,000,000 but codepen thought I had an infinite loop.

Note: the keyboard interaction doesn't seem to work in codepen

## Attributes

| Attribute | Description
|----       |----
| tabindex  | This needs to be set to enable keyboard interaction, the up and down keys only work when the grid has focus
| data      | Your collection of data (an array of objects)
| height    | The height of the grid in pixels
| rowheight | The height of each row in the grid in pixels (this defaults to 30)
| click     | Callback fired when a row is click, this returns the selcted row
| dblclick  | Callback fired when a row is double clicked, this returns the row selected (useful for edit events)
| onchange  | Callback fires whenever row selection changes. This returns and array of active rows. This fires on both mouse and keyboard events
| active    | Setting an array of rows will select them in the grid
| selectkey | optional keycode to select the row, ie 13 = enter

## Keyboard

* Clicking on the grid gives it keyboard focus, allowing multiple grids to work on the page at once.
* Arrow up selects the previous row in the grid
* Arrow down selects the next row
* Provider optional selectkey (see attributes above)
* Shift + Arrow adds more to a selection
* Shift + Click selects all rows between the first click, and the new click
* CMD/CTRL + Click toggles row selection

Note: during arrow up/down I scroll the panel to keep the selected row in the center of the panel.
This may need a little work as it sometimes jump to the center.

## About

I developed this because there doesn't seem to be any grid components available for riot (that I've found). This only
renders visible rows to the dom so is capable of handling large amounts of data. I've tried it with a million+ rows.
The browser tends to slow down because of the data processing before is slows down with the rendering.

The api may seem a little verbose, but being able to pass in the header and body rows gives a lot of flexibility
in terms of the data you choose to display. You can even include your own tags in the rows.

Each row has to be the same height. This may seem a little restrictive but it makes calculating which rows to render
much quicker.

## Reordering, Resizing, Data formats etc
The grid itself doesn't help with reordering, resizing or field formats. However the api is flexible enough
to allow you to easily implemnt this outside the grid component. This was a deliberate choice as I
want the grid to do one thing well, which is to allow you to scroll lots of data. I included the selection api
too as this is much easier to deal with when embedded.

## Style
The style applied to this grid is deliberately basic. Its intended to lay things out, but should be overriden
with your own colour scheme and style. You can probably work out the selectors used by using your web-inspector.
Though I may add some notes here in the future.

## Todo

* Add demo of reorder by column
* Add demo of change column width

## License

(The MIT License)

Copyright (c) 2015 Cris Ward

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repository: rongxike/riot-grid

repo_name: rongxike/riot-grid


text:
{fpl}
=======================================================================================================================

[![CRAN
status](https://www.r-pkg.org/badges/version/fpl)](https://cran.r-project.org/package=fpl)
[![Dependencies](https://tinyverse.netlify.com/badge/fpl)](https://cran.r-project.org/package=fpl)
![CRAN downloads](https://cranlogs.r-pkg.org/badges/fpl)
![check_cran](https://github.com/nathaneastwood/fpl/workflows/check_cran/badge.svg?branch=master)

Overview
--------

{fpl} is an {R6} wrapper around the Barclays Fantasy Premier League API.

Installation
------------

You can install:

-   the development version from
    [GitHub](https://github.com/nathaneastwood/fpl) with

``` r
# install.packages("remotes")
remotes::install_github("nathaneastwood/fpl")
```

Usage
-----

``` r
library(fpl)

fpl <- FPL$new()
team <- fpl$get_team(11)
team_players <- team$get_players()
```

Related Work
------------

-   [fpl](https://github.com/amosbastian/fpl) - a Python FPL API
    wrapper.
-   [{fplscrapR}](https://github.com/wiscostret/fplscrapR) - A
    functional FPL API wrapper.

repository: nathaneastwood/fpl

repo_name: nathaneastwood/fpl


text:
# FEDOT
[![Build Status](https://travis-ci.com/nccr-itmo/FEDOT.svg?token=ABTJ8bEXZokRxF3wLrtJ&branch=master)](https://travis-ci.com/nccr-itmo/FEDOT) [![Coverage Status](https://coveralls.io/repos/github/nccr-itmo/FEDOT/badge.svg?branch=master)](https://coveralls.io/github/nccr-itmo/FEDOT?branch=master)

This repository contains the framework for the knowledge-enriched AutoML named FEDOT (Russian: Федот).
It can be used to generate high-quality composite models for the classification, regression, clustering, time series forecasting, and other real-world problems in an automated way.

The project is maintained by the research team of Natural Systems Simulation Lab, which is a part of the National Center for Cognitive Research of ITMO University.

## Short description

To combine the proposed concepts and methods with the existing state-of-the-art approaches and share the obtained experience with the community, we decided to develop the FEDOT framework.

The framework kernel can be configured for different classes of tasks. The framework includes the library with implementations of intelligent algorithms to identify data-driven models with different requirements; and composite models (chains of models) for solving specific subject tasks (social and financial, metocean, physical, etc.).

It is possible to obtain models with given parameters of quality, complexity, interpretability; to get an any-time result; to pause and resume model identification; to integrate many popular Python open source solutions for AutoML/meta-learning, optimization, quality assessment, etc.; re-use the models created by other users.

## Documentation

The documentation is available in [FEDOT.Docs](https://itmo-nss-team.github.io/FEDOT.Docs) repository.

The description and source code of underlying algorithms is available in [FEDOT.Algs](https://github.com/ITMO-NSS-team/FEDOT.Algs) repository and its [wiki pages](https://github.com/ITMO-NSS-team/FEDOT.Algs/wiki) (in Russian).


text:
---
layout: post
comments: true
title: SQL Style for Data Scientists - 3 Simple Rules
subtitle: 3 Simple Rules for Writing Better SQL Code
excerpt: Here are some general rules that I try to follow when coding in a SQL  environment. There will always be exceptions, but you might want to keep these in mind when you're developing your own SQL scripts.
image: images/sql-style-guidelines-general-rules.png
date: 2018-10-04
author: pjryan126
category: techblog
tags: [sql, coding, style]
---

Following on last week's brief **[introduction](https://pjryan126.github.io/sql-style-intro/)**,
here are some general rules that I try to follow when coding in a SQL
environment. These rules are derived from Joe Celko's **[SQL Programming
Style](https://www.amazon.com/Celkos-Programming-Kaufmann-Management-Systems/dp/0120887975/ref=sr\_1\_1?ie=UTF8&qid=1516645111&sr=8-1&keywords=Joe+Celko+SQL+Programming+Style)**
and also appear in Simon Holywell's
**[SQL Style Guide](http://www.sqlstyle.guide/)**. There will always be
exceptions, but you might want to keep these rules in mind when you're
developing your own SQL scripts.

 It is your
opportunity to tell others in the clearest terms possible what your code should
do. In fact, it is often helpful to write your comments first before you event
start to write your query or function. It always helps to know where you're
going before you start your journey.

Generally speaking, you should try to use the C-style `/*` and `*/` marks to
open and close your comments whenever possible. Doing so will allow you to write
multi-line comments without worrying about adding a `--` mark to every single
line. If you are adding a quick comment on the same line as your code, though,
you can use a `--` mark and finish with a newline.

```
/* Update North Korea's country name */
UPDATE Tools.Countries
   SET country_name = 'Little Rocket Land' -- use the unofficial name
 WHERE country_id = 113;
```

Comment style is a matter of preference. The key is that you do it consistently
and in as a consistent a fashion as possible.

## Use White Space to Improve Readability {#white-space}

Do not, under any circumstances, write code like you are writing a paragraph.
Your code should be broken up into logical parts. Clauses, for example, should
appear on separate lines of code.

Below is an example of a simple `SELECT` query that uses white space and
indentation judiciously.

```
SELECT
       S.state_name   AS state,
       C.country_name AS country,
       C.iso3         AS country_code
  FROM Tools.States AS S
  LEFT OUTER JOIN Tools.Countries AS C
    ON S.country_id = C.country_id
 ORDER BY
       state,
       country;
```

## Use Standard SQL Functions {#standard-functions}

To maximize the portability of your code, you should do your best to use
standard SQL functions whenever possible. Every database management system puts
its own spin on SQL. In SQL Server, for example, you can replace `NULL` values
in a query with a value of your choosing with the `ISNULL()` function. However,
that function is not available anywhere outside SQL Server, so if you write a
query in SQL Server using the `ISNULL()` function and then try to run that
query on a MySQL database, you are going to get an error. A better choice would
be the `COALESCE()` function, which is available in all database management
systems.

```
SELECT
       country_id,
       iso,
       country_name,
       COALESCE(iso3, 'Not Available')        AS iso3,
       COALESCE(number_code, 'Not Available') AS number_code,
       phone_code
  FROM Tools.Countries
 WHERE iso <> 'US';
```

You can find a useful summary of standard and dialect-specific functions
**[here](https://en.wikibooks.org/wiki/SQL_Dialects_Reference)**. Many of the
system-specific functions can be quite useful, so I wouldn't say their use
should be expressly forbidden. If you are writing code that you think might be
used on more than one type of database management system, though, you should do
your best to avoid them.

{% if page.comments %}

If you have any ideas on how to improve these guidelines, please feel free to leave a comment below. I'd love to hear about it.

Please enable JavaScript to view the comments powered by Disqus.

{% endif %}


text:
# Change Log

All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.

## [0.0.60](https://github.com/johnsonandjohnson/bodiless-js/compare/v0.0.58...v0.0.60) (2020-11-18)

### Features

* **fclasses:** Design Keys ([#685](https://github.com/johnsonandjohnson/bodiless-js/issues/685)) ([0db060a](https://github.com/johnsonandjohnson/bodiless-js/commit/0db060ad1e7496553cd8eae3654770530d543010)), closes [#660](https://github.com/johnsonandjohnson/bodiless-js/issues/660)
* **search:** component library search component ([#592](https://github.com/johnsonandjohnson/bodiless-js/issues/592)) ([faa2219](https://github.com/johnsonandjohnson/bodiless-js/commit/faa2219a5dc5b66cbb3fd2a8eba6d24a04d3d38f))

## [0.0.59](https://github.com/johnsonandjohnson/bodiless-js/compare/v0.0.58...v0.0.59) (2020-11-05)

### Features

* **search:** component library search component ([#592](https://github.com/johnsonandjohnson/bodiless-js/issues/592)) ([faa2219](https://github.com/johnsonandjohnson/bodiless-js/commit/faa2219a5dc5b66cbb3fd2a8eba6d24a04d3d38f))

repository: akorachkina/Bodiless-JS

repo_name: akorachkina/Bodiless-JS


text:
---
layout: news
title: Welcome Jorge
date: 2017-02-02
excerpt: Welcome to our new group member Jorge Roel
tags: [HADDOCK, Utrecht University, Alexandre Bonvin, Docking]
image:
  feature:
---
Welcome to our new group member Jorge Roel, who is joining the HADDOCK team as a PhD candidate under the NWO TOP-PUNT project *"Caught in the act: a combined magnetic resonance and modelling approach to capture cellular machines at work"* (read about it [here](http://www.bonvinlab.org/news/TOP-PUNT-NWO-proposal-granted/)).

<figure align="center">
    <img src="/images/people/Jorge.png">
</figure>

Jorge did his master at the [Universitat Pompeu Fabra](https://www.upf.edu/en) in Barcelona and has been working in the group of [Dr. Juan Fernandez-Recio](http://mmb.pcb.ub.es/~juan/), a well-known expert in the docking field.

gha_language: HTML


text:
# Continuous Deliver for AI and ML Applications

This repository contains the sample application and machine learning code used for
the Continuous Delivery for AI & Machine Learning Applications (CDAMLA)
.

## Pre-Requisites

In order to run this , you will need:

* A valid Github account
* A working Docker setup (if running on Windows, make sure to use Linux containers)

## Instructions

The setup is divided into several steps, which build on top of each other.
Instructions for each can be found under the
[`instructions`](./instructions) folder.


text:
# types-angularjs-dragula

## Summary

This package contains TypeScript type definitions for [angularjs-dragula](https://github.com/bevacqua/angularjs-dragula).

## Installation

This is an unofficial package, so in order to install it under `@types` namespace it has to be installed from an archive rather that npm registry.

```
npm install --save https://github.com/xemlock/types-angularjs-dragula/tarball/v1.1.0
```

repository: xemlock/types-angularjs-dragula

repo_name: xemlock/types-angularjs-dragula


text:
<!-- TITLE: CTF solving using radare2 / blogs -->
# SecurisecCTF
SecurisecCTF is looking for new members! We are an open to all Slack community focused on all aspects of information security, and CTF team! Anyone with any skill level is welcome to join. Our goal is to teach/learn from each other. #ctf #infosec http://www.securisec.com *This is not a radare2 group, or is not affiliated with radare2 in anyway* however we do promote healthy discussions on reverse enginnering!

# CTF solving using radare2 / Blogs

- [A journey into Radare 2 - Part 1](https://www.megabeets.net/a-journey-into-radare-2-part-1/)

- [A journey into Radare 2 - Part 2: Exploitation - Megabeets](https://www.megabeets.net/a-journey-into-radare-2-part-2/)

- [Tools! Tools! Tools! Capitol City's solvingest binaries. They're not even obfuscating their bytes.](https://fevral.github.io/2017/08/13/flareon2015-2.html)

- [BinaryStudio](https://binarystud.io/googlectf-2017-inst-prof-152-final-value.html)

	- > *Google CTF 2017 qualifier PWN challenge*

- [Reverse TUCTF - Unknown](https://teamrocketist.github.io/2017/11/27/Reverse-TUCTF-Unknown/)

	- > *TUCTF 2017 Unknown writeup. This is an example of brute forcing strings based on a returned true statement using by patching on the fly*

- [Write-ups from RHME3 pre-qualifications at RADARE2 conference - Riscure](https://www.riscure.com/blog/write-ups-rhme3-pre-qualifications-radare2-conference/)
	- > _Three different writeups for the same challenge all using radare2_

- [ROPEmporium: Pivot 32-bit CTF Walkthrough With Radare2](http://radiofreerobotron.net/blog/2017/11/23/ropemporium-pivot-ctf-walkthrough/)

- [Reverse Engineering With Radare2, Part 1 | Sam Symons](https://samsymons.com/blog/reverse-engineering-with-radare2-part-1/)

- [Defeating ioli with radare2](https://dustri.org/b/defeating-ioli-with-radare2.html)

- [CMU Binary bomb solving with radare2](https://unlogic.co.uk/2016/04/14/Binary%20Bomb%20with%20Radare2%20-%20Phase%201/index.html)
- [CMU Binary bomb phase 2 by ctfsolver](http://ctfhacker.com/ctf/python/symbolic/execution/reverse/radare/2015/11/28/cmu-binary-bomb-flag2.html)
	- > Using radare2 and angr to solve the challenge
- [Solving game2 from the badge of Black Alps 2017 with radare2](https://dustri.org/b/solving-game2-from-the-badge-of-black-alps-2017-with-radare2.html)
- [TUMCTF Zwiebel](https://losfuzzys.github.io/writeup/2016/10/03/tumctf-zwiebel50/)
- [Rootcon 2017 binforcry 350](https://medium.com/@monliclican/rootcon-2017-ctf-binforcry-350-write-up-walkthrough-part-2-of-x-5731c91c2266)
	- > Writeup using r2 is halfway into the post

- [34c4 m0rph](https://www.sigflag.at/blog/2017/writeup-34c3ctf-m0rph/)
- [Intro to r2 Insinuator](https://insinuator.net/2016/08/reverse-engineering-with-radare2-intro/)
- [Linux Malware Analysis — Why Homebrew Encryption is Bad](https://goggleheadedhacker.com/blog/post/4)
- [Redvelvet codegate 2017 r2+angr+ida :(](https://github.com/Brandon-Everhart/CTF-Writeups/tree/master/2018/Codegate/Reversing/RedVelvet)
- [Ioli crackme writeups (various)](https://moveax.me/crackme0x06/)
- [Xiomara 2018 slammer](https://jbzteam.github.io/xiomaractf2018/Slammer)
	- [Another version of the Xiomara 2018 slammer](http://www.aperikube.fr/docs/xiomara_re4/)
- [TAMUCTF 18 Cutter](https://adminadminctf.github.io/ctf/TAMUctf18-hashbrowns)
- [Harekaze 2018 EzReverse](https://github.com/RedEcho/CTF_Challenges/blob/master/EasyCTF_IV/5365_EzReverse/WriteUp/Writeup.md)
- [Self modifying binary](https://www.megabeets.net/reversing-a-self-modifying-binary-with-radare2/)
- [Solving bomblab with r2](https://moveax.me/bomblab/)
- [Gynvael mission 22](https://ctfs.ghost.io/gynvael-mission-22-solution/amp/?__twitter_impression=true)
- [Bomblab moveax](https://moveax.me/bomblab/)
- [ROPping to victory](https://jmpesp.me/rop-emporium-ret2win-with-radare-and-pwntools/)
- 📼 [Solving a Self-modifying Crackme with r2pipe EMU vs DBG vs XOR](https://www.youtube.com/watch?v=ODCNbM9_U4M)
- [Reversing MalwareTech challenge with Radare2 and inline assembly](https://prsecurity.org/reversing-malwaretech-challange-with-radare2-and-inline-assembly/)
- [ROPping to Victory](https://jmpesp.me/rop-emporium-ret2win-with-radare-and-pwntools/)
- [smcauth defcon qual 2018](https://upbhack.de/posts/defcon-ctf-qualifier-2018-writeup-smcauth/)
- [Emulating decryption function with radare2](http://www.mien.in/2018/08/14/emulating-decryption-function-with-radare2/)
- [Hackcon-2018-WarChange](https://zenhack.it/writeups/HackCon2018/2018-08-17-Hackcon-2018-WarChange/)


text:
mbtiles-mapper
==============

A client-side [mbtiles](https://github.com/mapbox/mbtiles-spec) mapper for mbtiles uncompressed file.
Supports utfgrid and reads all metadata created by [Tilemil](https://www.mapbox.com/tilemill/).

Usage
=====

* Git clone this repository

* Uncompress your mbtiles file using [mbutil](https://github.com/mapbox/mbutil) with the `--grid_callback=""` option:

```
mb-util --grid_callback="" your_file.mbtiles destination_folder
```

* Modify `tiles_path` option in `config.json` with the destination folder path

* Start a static server to serve your map. For example, with [http-server](https://www.npmjs.org/package/http-server), go to your map directory and run:

```
http-server ./
```

* So you have your map in `http://localhost:8080`

Options
-------

Tiles path can be specified with params: `?tiles=path_to_tiles`.

Options can be modified in `config.json`. The `use_mbtiles_legend` option allows you to toggle between Tilemil generated legend and the one
created by you in index.html. Path specified in `tiles_path` option is loaded by default if `tiles` param is not present.

Libraries
=========

In the `lib` folder there are the needed libraries:

* [Jquery](http://jquery.com/)
* [Leaflet](https://github.com/Leaflet/Leaflet)
* [utfgrid.leaflet](https://github.com/danzel/Leaflet.utfgrid)

License
=======
MIT

repository: BrunoSalerno/mbtiles-mapper

repo_name: BrunoSalerno/mbtiles-mapper


text:
### Handshake
The handshake is required before any requests can be served.  If the first message sent cannot be parsed as a handshake, the connection will be dropped.  The handshake will be used to associate the client with a specific user (and set of security rules) on the server.  This should be extensible in the same way as #12.

For now let's just leave this a placeholder, since we haven't gotten to authentication yet.

#### Handshake Request
```
{
  "request_id": <NUMBER>,
  "method": "unauthenticated" | "anonymous" | "token",
  "token": <STRING>,
}
```

* `request_id` is a number uniquely identifying this request, it will be returned in the response.
* `method` designates the type of authentication to be performed.
  * `unauthenticated` performs no further steps and will not associate the connection with any user.
  * `anonymous` will create a new account with no external authentication provider.
  * `token` will associate the connection with the user in the horizon access token provided.
* `token` is the horizon access token that the client must already possess.
  * This field is required when `method` is `token`, and invalid otherwise.

#### Handshake Response
```
{
  "request_id": <NUMBER>,
  "token": <STRING>
}
```
* `token` is the horizon access token that is associated with this connection.
  * This token may be used to establish new connections under the same user account until the token expires.

#### Handshake Error Response
```
{
  "request_id": <NUMBER>,
  "error": <STRING>,
  "error_code": <NUMBER>
}
```

### Requests

All requests match the following pattern:
```
{
  "request_id": <NUMBER>,
  "type": <STRING>,
  "options": <OBJECT>
}
```
* `request_id` is a number uniquely identifying this request, it will be returned in any responses
* `type` is the endpoint for the query - one of `query`, `subscribe`, `store_error`, `store_replace`, `update`, or `remove`.
* `options` is an object structured differently for each endpoint.

#### query, subscribe

```
{
  "request_id": <NUMBER>,
  "type": "query" | "subscribe",
  "options": {
    "collection": <STRING>,
    "order": [ <ARRAY>, "ascending" | "descending"],
    "above": [ <OBJECT>, "open" | "closed" ],
    "below": [ <OBJECT>, "open" | "closed" ],
    "find": <OBJECT>,
    "find_all": [<OBJECT>, ...],
    "limit": <NUMBER>,
  }
}
```
* `collection` describes which table to operate on in the horizon database.
* `order` orders the results according to an array of fields - optional.
  * The first argument is an array of field names, most-significant first.
  * The second argument determines which direction the results are sorted in.
* `above` and `below` are arrays describing the boundaries regarding `order` - optional.
  * `above` and `below` can only be specified if `order` is provided.
  * The first argument is an object whose key-value pairs correspond to fields in `order`.
  * The second argument should be `closed` to include the boundary, and `open` otherwise.
* `find` returns one object in `collection` that exactly matches the fields in the object given - optional.
  * `find` cannot be used with `find_all`, `order`, `above`, or `below`.
* `find_all` is an array of objects whose key-value pairs correspond to keys in `index` - optional.
  * Returns any object in `collection` that exactly matches the fields in any of the objects given.
  * `find_all` cannot be used with `find`.
  * `find_all` with multiple objects cannot be used with `order`, `above`, or `below`.
* `limit` limits the number of results to be selected - optional.

#### insert, store, upsert, replace, update, remove

```
{
  "request_id": <NUMBER>,
  "type": "store" | "update" | "upsert" | "insert" | "replace" | "remove",
  "options": {
    "collection": <STRING>,
    "data": [<OBJECT>, ... ]
  }
}
```
* `collection` describes which table to operate on in the horizon database
* `data` is the documents to be written (or removed)
  * `data[i].id` is required for `remove` operations, all other fields are optional
  * `data[i].id` may be omitted in an `insert`, `store`, or `upsert` operations: a new row will be inserted in the collection
* `type` is the write operation to perform
  * `insert` inserts new documents, erroring if any document already exists
  * `update` updates existing documents. It errors if any document does not already exist
  * `upsert` updates existing documents or inserts them if they do not exist
  * `replace` replaces existing documents entirely. It errors if any document does not already exist
  * `store` replaces existing documents entirely, or inserts them if they don't exist.
  * `remove` removes documents. It will not error if a document does not exist

#### end_subscription
Tells the horizon server to stop sending data for a given subscription.  Data may still be received until the server has processed this and sent a `"state": "complete"` response for the subscription.
```
{
  "request_id": <NUMBER>,
  "type": "end_subscription"
}
```

#### Keepalive
This is used by the client to perform an empty request to avoid connection interruption.
```
{
  "request_id": <NUMBER>,
  "type": "keepalive"
}
```

### Responses

#### Error Response
This can be sent for any request at any time.  Once an error response is sent, no further responses shall be sent for the corresponding `request_id`.
```
{
  "request_id": <NUMBER>,
  "error": <STRING>,
  "error_code": <INTEGER>
}
```
* `request_id` is the same as the `request_id` in the corresponding request
* `error` is a descriptive error string
* `error_code` is a code that can be used to identify the type of error, values TBD

#### query, subscribe
`query` and `subscribe` requests will result in a stream of results from the horizon server to the client.  The stream will be an ordered set of messages from the server following the structure below.  If an error occurs, the above Error Response structure will be used, and the stream is considered "complete".  An Error Response may still be sent even after a successful data response, but not after `"state": "complete"`.
```
{
  "request_id": <NUMBER>,
  "data": <ARRAY>,
  "state": "synced" | "complete"
}
```
* `request_id` is the same as the `request_id` in the corresponding request
* `data` is an array of results for the `query` or `subscribe`, and may be empty
* `state` is optional, and indicates a change in the stream of data:
  * `synced` means that following the consumption of `data`, the client has all the initial results of a `subscribe`
  * `complete` means that following the consumption of `data`, no more results will be returned for the request

#### Write responses
`store`, `replace`, `insert`, `update`, `upsert`, and `remove` requests will be given a single response.  This may be an Error Response, or:
```
{
  "request_id": <NUMBER>,
  "data": [ { "id": <DOCUMENT_ID>, "$hz_v$": <DOCUMENT_VERSION> } | { "error": <STRING>, "error_code": <INTEGER> }, ...],
  "state": "complete"
}
```
* `data` is an array of objects corresponding to the documents specified in the write (whether or not a change occurred). For inserted documents it will be the id generated by the server as well as the latest version field for the affected document.  If an error occurred, there will instead be an error description string and an error code in the object .  The items in the array correspond directly to the changes in the request, in the same order.
* `state` can only be "complete" for write responses

#### Keepalive
`keepalive` requests will be given a single response.  This will never be an error response unless there is a protocol error.
```
{
  "request_id": <NUMBER>,
  "state": "complete"
}
```

repository: ramonbrugman/horizon

branch_name: refs/heads/next

repo_name: ramonbrugman/horizon


text:
Online payment is a critical part of any ecommerce solution. This module makes use of the PHP Omnipay payment library, which does a great job of standardising online payments.

This module integrates with the omnipay library via the [SilverStripe-Omnipay module](https://github.com/silverstripe/silverstripe-omnipay). It is automatically required by the shop module, via composer.

This Omnipay presentation gives a general overview to online payments and integrating omnipay with SilverStripe: http://jeremyshipman.com/blog/my-2c-on-omnipay-integrating-with-silverstripe/

## Available payment types

There are many [gateways](https://github.com/thephpleague/omnipay#payment-gateways) available, which you can install separately. Note that currently this module uses version 2.x of the Omnipay library.

The best way to find additional Omnipay payment drivers is perhaps to do a search on Packagist: https://packagist.org/search/?q=omnipay

Here is a tutorial for [setting up PayPal via Omnipay](https://github.com/silverstripe/silverstripe-omnipay/blob/master/docs/en/PayPalExpressSetup.md).

repository: AntonyThorpe/silvershop-core

repo_name: AntonyThorpe/silvershop-core