module
support:community
bug
traceback
affects_2.7
crypto

acme_certificate - dest must include path info or fails

<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Explain the problem briefly below -->
Calling acme_certificate with dest set to a pure filename (no path) will cause the module to fail.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
acme_certificate

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible 2.7.9
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/dhagan/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/dhagan/.local/lib/python2.7/site-packages/ansible
  executable location = /home/dhagan/.local/bin/ansible
  python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
DEFAULT_CALLBACK_WHITELIST(/etc/ansible/ansible.cfg) = [u'timer']
```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
Ubuntu 18.04 on Windows 10

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->
Example below uses dns-01 challenge against route53.  Either fill in w/ appropriate domain info, or modify as you see fit to allow challenge to pass.

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: letsencrypt
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    target: "myhost.mytld.com"
    zone: "mytld.com"
    contact: "mailto:name@example.com"

  tasks:
    - name: acme account
      acme_account:
        account_key_src: test_key.pem
        acme_directory: "https://acme-staging-v02.api.letsencrypt.org/directory"
        acme_version: 2
        allow_creation: true
        contact: "{{ contact }}"
        state: present
        terms_agreed: yes
        validate_certs: yes
      register: account

    - name: create private key
      openssl_privatekey:
        path: test.key
        size: 2048
        type: RSA

    - name: create CSR if not present
      openssl_csr:
        common_name: "{{ target }}"
        path: test.csr
        privatekey_path: test.key
        subject_alt_name: "DNS:{{target}}"

    - name: acme request
      acme_certificate:
        account_key_src: test_key.pem
        modify_account: no
        account_uri: "{{ account.account_uri }}"
        challenge: "dns-01"
        csr: test.csr
        dest: test.cert
        terms_agreed: yes
        validate_certs: yes
      register: acme_request

    - name: meet challenge requirements
      route53:
        zone: "{{ zone }}"
        record: "{{ acme_request.challenge_data[target]['dns-01'].record }}"
        type: TXT
        ttl: 60
        state: present
        overwrite: yes
        wait: yes
        value: "{{ acme_request.challenge_data[target]['dns-01'].resource_value | regex_replace('^(.*)$', '\"\\1\"') }}"
      when: acme_request is changed

    - name: acme certificate
      acme_certificate:
        account_key_src: test_key.pem
        modify_account: no
        account_uri: "{{ account.account_uri }}"
        challenge: "dns-01"
        src: test.csr
        dest: test.cert
        data: "{{ acme_request }}"

```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
Certificate gets written to test.cert in current directory.

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
This section of ansible/lib/ansible/module_utils/acme.py, starting at line 121, causes the module to fail because os.path.dirname(dest) for a bare filename is empty.

```
else:
        if not os.access(os.path.dirname(dest), os.W_OK):
            os.remove(tmpsrc)
            raise ModuleFailException("Destination dir %s not writable" % (os.path.dirname(dest)))
```
<!--- Paste verbatim command output between quotes -->
```paste below
dhagan@onmyoji-shi:~/cloud-ansible$ ansible-playbook -vvv test.yml
ansible-playbook 2.7.9
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/dhagan/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/dhagan/.local/lib/python2.7/site-packages/ansible
  executable location = /home/dhagan/.local/bin/ansible-playbook
  python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]
Using /etc/ansible/ansible.cfg as config file
/etc/ansible/hosts did not meet host_list requirements, check plugin documentation if this is unexpected
/etc/ansible/hosts did not meet script requirements, check plugin documentation if this is unexpected
Parsed /etc/ansible/hosts inventory source with ini plugin
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAYBOOK: test.yml **************************************************************************************************************************************************************************************************************************
1 plays in test.yml

PLAY [letsencrypt] **************************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [acme account] *************************************************************************************************************************************************************************************************************************
task path: /mnt/xxxx/test.yml:12
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: dhagan
<127.0.0.1> EXEC /bin/sh -c 'echo ~dhagan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239227.37-238894588553222 `" && echo ansible-tmp-1554239227.37-238894588553222="` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239227.37-238894588553222 `" ) && sleep 0'
Using module file /home/dhagan/.local/lib/python2.7/site-packages/ansible/modules/crypto/acme/acme_account.py
<127.0.0.1> PUT /home/dhagan/.ansible/tmp/ansible-local-12205ml_9CV/tmpmJ9dFJ TO /home/dhagan/.ansible/tmp/ansible-tmp-1554239227.37-238894588553222/AnsiballZ_acme_account.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/dhagan/.ansible/tmp/ansible-tmp-1554239227.37-238894588553222/ /home/dhagan/.ansible/tmp/ansible-tmp-1554239227.37-238894588553222/AnsiballZ_acme_account.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/dhagan/.ansible/tmp/ansible-tmp-1554239227.37-238894588553222/AnsiballZ_acme_account.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/dhagan/.ansible/tmp/ansible-tmp-1554239227.37-238894588553222/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "account_uri": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/xxxx",
    "changed": false,
    "invocation": {
        "module_args": {
            "account_key_content": null,
            "account_key_src": "test_key.pem",
            "account_uri": null,
            "acme_directory": "https://acme-staging-v02.api.letsencrypt.org/directory",
            "acme_version": 2,
            "allow_creation": true,
            "contact": [
                "mailto:xxxxxx"
            ],
            "new_account_key_content": null,
            "new_account_key_src": null,
            "select_crypto_backend": "auto",
            "state": "present",
            "terms_agreed": true,
            "validate_certs": true
        }
    }
}

TASK [create private key] *******************************************************************************************************************************************************************************************************************
task path: /xxxx/test.yml:24
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: dhagan
<127.0.0.1> EXEC /bin/sh -c 'echo ~dhagan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239231.41-236536755185355 `" && echo ansible-tmp-1554239231.41-236536755185355="` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239231.41-236536755185355 `" ) && sleep 0'
Using module file /home/dhagan/.local/lib/python2.7/site-packages/ansible/modules/crypto/openssl_privatekey.py
<127.0.0.1> PUT /home/dhagan/.ansible/tmp/ansible-local-12205ml_9CV/tmpD_PkzS TO /home/dhagan/.ansible/tmp/ansible-tmp-1554239231.41-236536755185355/AnsiballZ_openssl_privatekey.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/dhagan/.ansible/tmp/ansible-tmp-1554239231.41-236536755185355/ /home/dhagan/.ansible/tmp/ansible-tmp-1554239231.41-236536755185355/AnsiballZ_openssl_privatekey.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/dhagan/.ansible/tmp/ansible-tmp-1554239231.41-236536755185355/AnsiballZ_openssl_privatekey.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/dhagan/.ansible/tmp/ansible-tmp-1554239231.41-236536755185355/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false,
    "filename": "test.key",
    "fingerprint": {
        "md5": "xxxx",
        "sha1": "xxxx",
        "sha224": "xxxx",
        "sha256": "xxxx",
        "sha384": "xxxx",
        "sha512": "xxxx"
    },
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": null,
            "cipher": null,
            "content": null,
            "delimiter": null,
            "directory_mode": null,
            "follow": false,
            "force": false,
            "group": null,
            "mode": null,
            "owner": null,
            "passphrase": null,
            "path": "test.key",
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "size": 2048,
            "src": null,
            "state": "present",
            "type": "RSA",
            "unsafe_writes": null
        }
    },
    "size": 2048,
    "type": "RSA"
}

TASK [create CSR if not present] ************************************************************************************************************************************************************************************************************
task path: /xxxx/test.yml:30
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: dhagan
<127.0.0.1> EXEC /bin/sh -c 'echo ~dhagan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239232.48-242780450892665 `" && echo ansible-tmp-1554239232.48-242780450892665="` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239232.48-242780450892665 `" ) && sleep 0'
Using module file /home/dhagan/.local/lib/python2.7/site-packages/ansible/modules/crypto/openssl_csr.py
<127.0.0.1> PUT /home/dhagan/.ansible/tmp/ansible-local-12205ml_9CV/tmpOlkhLU TO /home/dhagan/.ansible/tmp/ansible-tmp-1554239232.48-242780450892665/AnsiballZ_openssl_csr.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/dhagan/.ansible/tmp/ansible-tmp-1554239232.48-242780450892665/ /home/dhagan/.ansible/tmp/ansible-tmp-1554239232.48-242780450892665/AnsiballZ_openssl_csr.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/dhagan/.ansible/tmp/ansible-tmp-1554239232.48-242780450892665/AnsiballZ_openssl_csr.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/dhagan/.ansible/tmp/ansible-tmp-1554239232.48-242780450892665/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "basicConstraints": null,
    "changed": false,
    "extendedKeyUsage": null,
    "filename": "test.csr",
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": null,
            "basicConstraints": null,
            "basicConstraints_critical": false,
            "commonName": "xxxx",
            "common_name": "xxxx",
            "content": null,
            "countryName": null,
            "delimiter": null,
            "digest": "sha256",
            "directory_mode": null,
            "emailAddress": null,
            "extendedKeyUsage": null,
            "extendedKeyUsage_critical": false,
            "follow": false,
            "force": false,
            "group": null,
            "keyUsage": null,
            "keyUsage_critical": false,
            "localityName": null,
            "mode": null,
            "ocspMustStaple": false,
            "ocspMustStaple_critical": false,
            "organizationName": null,
            "organizationalUnitName": null,
            "owner": null,
            "path": "test.csr",
            "privatekey_passphrase": null,
            "privatekey_path": "test.key",
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": null,
            "state": "present",
            "stateOrProvinceName": null,
            "subject": null,
            "subjectAltName": [
                "DNS:xxxx"
            ],
            "subjectAltName_critical": false,
            "subject_alt_name": "DNS:xxxx",
            "unsafe_writes": null,
            "version": 1
        }
    },
    "keyUsage": null,
    "ocspMustStaple": false,
    "privatekey": "test.key",
    "subject": [
        [
            "CN",
            "xxxx"
        ]
    ],
    "subjectAltName": [
        "DNS:xxxx"
    ]
}

TASK [acme request] *************************************************************************************************************************************************************************************************************************
task path: /xxx/test.yml:37
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: dhagan
<127.0.0.1> EXEC /bin/sh -c 'echo ~dhagan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239233.51-137769391892495 `" && echo ansible-tmp-1554239233.51-137769391892495="` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239233.51-137769391892495 `" ) && sleep 0'
Using module file /home/dhagan/.local/lib/python2.7/site-packages/ansible/modules/crypto/acme/acme_certificate.py
<127.0.0.1> PUT /home/dhagan/.ansible/tmp/ansible-local-12205ml_9CV/tmpgY1bDo TO /home/dhagan/.ansible/tmp/ansible-tmp-1554239233.51-137769391892495/AnsiballZ_acme_certificate.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/dhagan/.ansible/tmp/ansible-tmp-1554239233.51-137769391892495/ /home/dhagan/.ansible/tmp/ansible-tmp-1554239233.51-137769391892495/AnsiballZ_acme_certificate.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/dhagan/.ansible/tmp/ansible-tmp-1554239233.51-137769391892495/AnsiballZ_acme_certificate.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/dhagan/.ansible/tmp/ansible-tmp-1554239233.51-137769391892495/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "account_uri": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/xxxx",
    "authorizations": {
        "xxxx": {
            "challenges": [
                {
                    "status": "pending",
                    "token": "xxxx",
                    "type": "dns-01",
                    "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/xxxx/xxxx"
                },
                {
                    "status": "pending",
                    "token": "xxxx",
                    "type": "tls-alpn-01",
                    "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/xxxx/xxxx"
                },
                {
                    "status": "pending",
                    "token": "xxxx",
                    "type": "http-01",
                    "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/xxxx/xxxx"
                }
            ],
            "combinations": [
                [
                    1
                ],
                [
                    2
                ],
                [
                    0
                ]
            ],
            "expires": "2019-04-09T21:07:15Z",
            "identifier": {
                "type": "dns",
                "value": "xxxx"
            },
            "status": "pending",
            "uri": "https://acme-staging.api.letsencrypt.org/acme/authz/xxxx"
        }
    },
    "cert_days": -1,
    "challenge_data": {
        "test-name.dhagan.dev.nsoc.state911.us": {
            "dns-01": {
                "record": "_acme-challenge.xxxx",
                "resource": "_acme-challenge",
                "resource_value": "xxxx"
            },
            "http-01": {
                "resource": ".well-known/acme-challenge/xxxx",
                "resource_value": "xxxx"
            },
            "tls-alpn-01": {
                "resource": "xxxx",
                "resource_value": "xxxx"
            }
        }
    },
    "challenge_data_dns": {
        "_acme-challenge.xxxx": [
            "xxxx"
        ]
    },
    "changed": true,
    "finalize_uri": null,
    "invocation": {
        "module_args": {
            "account_email": null,
            "account_key_content": null,
            "account_key_src": "test_key.pem",
            "account_uri": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/xxxx",
            "acme_directory": "https://acme-staging.api.letsencrypt.org/directory",
            "acme_version": 1,
            "agreement": null,
            "chain_dest": null,
            "challenge": "dns-01",
            "csr": "test.csr",
            "data": null,
            "deactivate_authzs": false,
            "dest": "test.cert",
            "force": false,
            "fullchain_dest": null,
            "modify_account": false,
            "remaining_days": 10,
            "select_crypto_backend": "auto",
            "terms_agreed": true,
            "validate_certs": true
        }
    },
    "order_uri": null
}

TASK [meet challenge requirements] **********************************************************************************************************************************************************************************************************
task path: /xxxx/test.yml:49
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: dhagan
<127.0.0.1> EXEC /bin/sh -c 'echo ~dhagan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239236.67-178715692795110 `" && echo ansible-tmp-1554239236.67-178715692795110="` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239236.67-178715692795110 `" ) && sleep 0'
Using module file /home/dhagan/.local/lib/python2.7/site-packages/ansible/modules/cloud/amazon/route53.py
<127.0.0.1> PUT /home/dhagan/.ansible/tmp/ansible-local-12205ml_9CV/tmpCRBtC3 TO /home/dhagan/.ansible/tmp/ansible-tmp-1554239236.67-178715692795110/AnsiballZ_route53.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/dhagan/.ansible/tmp/ansible-tmp-1554239236.67-178715692795110/ /home/dhagan/.ansible/tmp/ansible-tmp-1554239236.67-178715692795110/AnsiballZ_route53.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/dhagan/.ansible/tmp/ansible-tmp-1554239236.67-178715692795110/AnsiballZ_route53.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/dhagan/.ansible/tmp/ansible-tmp-1554239236.67-178715692795110/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "alias": null,
            "alias_evaluate_target_health": false,
            "alias_hosted_zone_id": null,
            "aws_access_key": null,
            "aws_secret_key": null,
            "ec2_url": null,
            "failover": null,
            "health_check": null,
            "hosted_zone_id": null,
            "identifier": null,
            "overwrite": true,
            "private_zone": false,
            "profile": null,
            "record": "_acme-challenge.xxxx",
            "region": null,
            "retry_interval": "500",
            "security_token": null,
            "state": "present",
            "ttl": 60,
            "type": "TXT",
            "validate_certs": true,
            "value": [
                "\"xxxx\""
            ],
            "vpc_id": null,
            "wait": true,
            "wait_timeout": 600,
            "weight": null,
            "zone": "xxxx"
        }
    }
}

TASK [acme certificate] *********************************************************************************************************************************************************************************************************************
task path: /xxxx/test.yml:62
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: dhagan
<127.0.0.1> EXEC /bin/sh -c 'echo ~dhagan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239274.52-105161436966429 `" && echo ansible-tmp-1554239274.52-105161436966429="` echo /home/dhagan/.ansible/tmp/ansible-tmp-1554239274.52-105161436966429 `" ) && sleep 0'
Using module file /home/dhagan/.local/lib/python2.7/site-packages/ansible/modules/crypto/acme/acme_certificate.py
<127.0.0.1> PUT /home/dhagan/.ansible/tmp/ansible-local-12205ml_9CV/tmpvcA5GB TO /home/dhagan/.ansible/tmp/ansible-tmp-1554239274.52-105161436966429/AnsiballZ_acme_certificate.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/dhagan/.ansible/tmp/ansible-tmp-1554239274.52-105161436966429/ /home/dhagan/.ansible/tmp/ansible-tmp-1554239274.52-105161436966429/AnsiballZ_acme_certificate.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/dhagan/.ansible/tmp/ansible-tmp-1554239274.52-105161436966429/AnsiballZ_acme_certificate.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/dhagan/.ansible/tmp/ansible-tmp-1554239274.52-105161436966429/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_acme_certificate_payload_B3NU6E/__main__.py", line 931, in main
    client.get_certificate()
  File "/tmp/ansible_acme_certificate_payload_B3NU6E/__main__.py", line 824, in get_certificate
    if self.dest and write_file(self.module, self.dest, pem_cert.encode('utf8')):
  File "/tmp/ansible_acme_certificate_payload_B3NU6E/ansible_acme_certificate_payload.zip/ansible/module_utils/acme.py", line 138, in write_file
    raise ModuleFailException("Destination dir %s not writable" % (os.path.dirname(dest)))

fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "account_email": null,
            "account_key_content": null,
            "account_key_src": "test_key.pem",
            "account_uri": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/xxxx",
            "acme_directory": "https://acme-staging.api.letsencrypt.org/directory",
            "acme_version": 1,
            "agreement": null,
            "chain_dest": null,
            "challenge": "dns-01",
            "csr": "test.csr",
            "data": {
                "account_uri": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/xxxx",
                "authorizations": {
                    "test-name.dhagan.dev.nsoc.state911.us": {
                        "challenges": [
                            {
                                "status": "pending",
                                "token": "xxxx",
                                "type": "dns-01",
                                "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/xxxx/xxxx"
                            },
                            {
                                "status": "pending",
                                "token": "xxxx",
                                "type": "tls-alpn-01",
                                "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/xxxxx/xxxx"
                            },
                            {
                                "status": "pending",
                                "token": "xxxx",
                                "type": "http-01",
                                "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/xxxx/xxxx"
                            }
                        ],
                        "combinations": [
                            [
                                1
                            ],
                            [
                                2
                            ],
                            [
                                0
                            ]
                        ],
                        "expires": "2019-04-09T21:07:15Z",
                        "identifier": {
                            "type": "dns",
                            "value": "xxxx"
                        },
                        "status": "pending",
                        "uri": "https://acme-staging.api.letsencrypt.org/acme/authz/xxxx"
                    }
                },
                "cert_days": -1,
                "challenge_data": {
                    "xxxx": {
                        "dns-01": {
                            "record": "_acme-challenge.xxxx",
                            "resource": "_acme-challenge",
                            "resource_value": "xxxx"
                        },
                        "http-01": {
                            "resource": ".well-known/acme-challenge/xxxx",
                            "resource_value": "xxxx"
                        },
                        "tls-alpn-01": {
                            "resource": "xxxx",
                            "resource_value": "xxxx"
                        }
                    }
                },
                "challenge_data_dns": {
                    "_acme-challenge.xxxx": [
                        "xxxx"
                    ]
                },
                "changed": true,
                "failed": false,
                "finalize_uri": null,
                "order_uri": null
            },
            "deactivate_authzs": false,
            "dest": "test.cert",
            "force": false,
            "fullchain_dest": null,
            "modify_account": false,
            "remaining_days": 10,
            "select_crypto_backend": "auto",
            "src": "test.csr",
            "terms_agreed": false,
            "validate_certs": true
        }
    },
    "msg": "Destination dir  not writable",
    "other": {}
}
        to retry, use: --limit @/xxxx/test.retry

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=5    changed=2    unreachable=0    failed=1

Playbook run took 0 days, 0 hours, 1 minutes, 0 seconds
```


How to fine tune EncoderDecoder model for training a new corpus of data ?

is there any documentation available for the same?


easyfix
python3
module
support:core
bug
has_pr
P3
system
affects_2.9

Reboot module doesn't work with async

<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
The `reboot` module does not work with `async`, `poll`, and `async_status`. Suppose I have 10 nodes to reboot, but I can only set `fork` to 2. The `reboot` module will reboot 2 nodes at a time. I tried using `async`, `poll`, and `async_status` to kick of the reboots on the 10 nodes, 2 at a time, and then poll for the results. `async` and `poll` seem to do nothing on the `reboot` module as the behavior remains the same as without them.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
`reboot` module

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible 2.9.12
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib64/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, May  2 2019, 19:37:42) [GCC 4.4.7 20120313 (Red Hat 4.4.7-23)]
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/etc/ansible/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null
ANSIBLE_SSH_RETRIES(/etc/ansible/ansible.cfg) = 2
COMMAND_WARNINGS(/etc/ansible/ansible.cfg) = False
DEFAULT_FORKS(/etc/ansible/ansible.cfg) = 2
DEFAULT_GATHERING(/etc/ansible/ansible.cfg) = explicit
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 40
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False
```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->
Described in the summary

<!--- Paste example playbooks or commands between quotes below -->
```yaml

```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
I expect the `reboot` module to start the reboot on 2 nodes, then move on to something else (like start the reboot on another 2 nodes), then come back to check on the results of the reboots by using `async`, `poll`, and `async_status`.

##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
The `reboot` module ignores `async` and `poll`.

<!--- Paste verbatim command output between quotes -->
```paste below

```

{'base_commit': '2908a2c32a81fca78277a22f15fa8e3abe75e092', 'files': [{'path': 'lib/ansible/plugins/action/reboot.py', 'status': 'modified', 'Loc': {"('ActionModule', 'run', 409)": {'mod': [411]}}}]}


HDF5: empty groups and keys

Hi,

With some of the hdf5 files I have, `pandas.HDFStore.groups()` returns an empty list. (as does `.keys()` which iterates over the groups). However, the data are accessible via `.get()` or `.get_node()`.

This is related to #21543 and #21372 where the `.groups()` logic was changed, in particular using `self._handle.walk_groups()` instead of `self._handle.walk_nodes()`, now to be found here:
https://github.com/pandas-dev/pandas/blob/ea2e26ae7d700d7fd363ea5bfc05d2fe3fb8a5ee/pandas/io/pytables.py#L1212

#### Current Output

```python
>>> hdf.groups()
[]
```
```python
>>> hdf.keys()
[]
```

#### Expected Ouptut

List of groups and keys as visible with e.g. `h5dump`.
**Note:** Changing the aforementioned line back to use `.walk_nodes()` fixes the issue and lists the groups and keys properly:

```python
>>> hdf.groups()
[/Data/Table Layout (Table(69462,), zlib(4)) ''
   description := {
...
/Data/Array Layout/2D Parameters/Data Parameters (Table(15,)) ''
   description := {
   "mnemonic": StringCol(itemsize=8, shape=(), dflt=b'', pos=0),
   "description": StringCol(itemsize=48, shape=(), dflt=b'', pos=1),
   "isError": Int64Col(shape=(), dflt=0, pos=2),
   "units": StringCol(itemsize=7, shape=(), dflt=b'', pos=3),
   "category": StringCol(itemsize=31, shape=(), dflt=b'', pos=4)}
   byteorder := 'little'
   chunkshape := (642,)]]
```
```python
>>> hdf.keys()
['/Data/Table Layout',
 '/Metadata/Data Parameters',
 '/Metadata/Experiment Notes',
 '/Metadata/Experiment Parameters',
 '/Metadata/Independent Spatial Parameters',
 '/Metadata/_record_layout',
 '/Data/Array Layout/Layout Description',
 '/Data/Array Layout/1D Parameters/Data Parameters',
 '/Data/Array Layout/2D Parameters/Data Parameters']
```

#### Fix

One solution would be (I guess) to revert #21543, another to fix at least `.keys()` to use `._handle.walk_nodes()` instead of `.groups()` in
https://github.com/pandas-dev/pandas/blob/ea2e26ae7d700d7fd363ea5bfc05d2fe3fb8a5ee/pandas/io/pytables.py#L562

Could also be that it is a bug in `pytables`.

#### Problem background

I was trying to figure out why some hdf5 files open fine with `pandas` but fail with `dask`.
The reason is that `dask` allows wildcards and iterates over the keys to find valid ones. If `.keys()` is empty, reading the files with `dask` fails.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Linux
OS-release       : 3.10.0-957.27.2.el7.x86_64
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : C
LOCALE           : en_US.UTF-8

pandas           : 0.25.3
numpy            : 1.17.3
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.3.1
setuptools       : 42.0.1.post20191125
Cython           : None
pytest           : 5.0.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.4.2
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : 7.10.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.2
matplotlib       : 3.1.2
numexpr          : 2.7.0
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.2
sqlalchemy       : None
tables           : 3.6.1
xarray           : 0.14.1
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>

{'base_commit': 'c482b5727e3bd98b6f9780e51615791e413d542d', 'files': [{'path': 'doc/source/whatsnew/v1.1.0.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [964]}}}, {'path': 'pandas/io/pytables.py', 'status': 'modified', 'Loc': {"('HDFStore', 'keys', 583)": {'add': [586, 590], 'mod': [592]}, "('HDFStore', None, 442)": {'mod': [583]}}}, {'path': 'pandas/tests/io/pytables/test_store.py', 'status': 'modified', 'Loc': {"('TestHDFStore', None, 66)": {'add': [343]}}}]}


List of fields incorrectly accessed for dataclass items

### Description

If I make a `dataclass` item and want to export to csv, I get this error:

```
...
  File "/home/tadej/miniconda3/envs/main/lib/python3.7/site-packages/scrapy/exporters.py", line 251, in _write_headers_and_set_fields_to_export
    self.fields_to_export = list(item.fields.keys())
AttributeError: 'CompanyItem' object has no attribute 'fields'
```
The problem stems from here

https://github.com/scrapy/scrapy/blob/master/scrapy/exporters.py#L243-L253

There should be an additional if case checking if the item is of type dataclass, and then accessing the fields differently, perhaps as
```python
[field.name for field in fields(item)]
```

{'base_commit': 'f3372a3753643fea601564c01fcf65cc25a2db62', 'files': [{'path': 'scrapy/exporters.py', 'status': 'modified', 'Loc': {"('CsvItemExporter', '_write_headers_and_set_fields_to_export', 243)": {'mod': [246, 247, 248, 249, 250, 251]}}}, {'path': 'tests/test_exporters.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [10, 25, 159, 168, 199, 221, 311, 414, 451, 529]}, "('BaseItemExporterTest', None, 26)": {'add': [27]}, "('BaseItemExporterTest', 'setUp', 28)": {'mod': [29]}, "('BaseItemExporterTest', '_assert_expected_item', 39)": {'mod': [42]}, "('BaseItemExporterTest', 'test_export_dict_item', 65)": {'mod': [66]}, "('BaseItemExporterTest', 'test_serialize_field', 68)": {'mod': [69, 72]}, "('BaseItemExporterTest', 'test_field_custom_serializer', 84)": {'mod': [85, 86, 88, 89, 90, 92, 94, 95, 96]}, "('PythonItemExporterTest', 'test_nested_item', 107)": {'mod': [108, 110]}, "('PythonItemExporterTest', 'test_export_list', 121)": {'mod': [122, 123, 124]}, "('PythonItemExporterTest', 'test_export_item_dict_list', 134)": {'mod': [135, 137]}, "('PythonItemExporterTest', 'test_export_binary', 147)": {'mod': [149]}, "('PickleItemExporterTest', 'test_export_multiple_items', 177)": {'mod': [178, 179, 187, 188]}, "('CsvItemExporterTest', 'test_header_export_all', 245)": {'mod': [248]}, "('CsvItemExporterTest', 'test_header_export_all_dict', 252)": {'mod': [254]}, "('CsvItemExporterTest', 'test_header_export_single_field', 258)": {'mod': [259]}, "('CsvItemExporterTest', 'test_header_export_two_items', 266)": {'mod': [267]}, "('CsvItemExporterTest', 'test_header_no_header_line', 277)": {'mod': [278]}, "('XmlItemExporterTest', 'xmltuple', 318)": {'mod': [321, 322]}, "('XmlItemExporterTest', 'test_multivalued_fields', 346)": {'mod': [348, 349, 350, 351, 352]}, "('XmlItemExporterTest', 'test_nested_item', 355)": {'mod': [356, 358]}, "('XmlItemExporterTest', 'test_nested_list_item', 378)": {'mod': [379, 381]}, "('JsonLinesItemExporterTest', '_check_output', 422)": {'mod': [424]}, "('JsonLinesItemExporterTest', 'test_nested_item', 426)": {'mod': [427, 429]}, "('JsonItemExporterTest', '_check_output', 459)": {'mod': [461]}, "('JsonItemExporterTest', 'assertTwoItemsExported', 463)": {'mod': [469]}, "('JsonItemExporterTest', 'test_two_dict_items', 474)": {'mod': [475]}, "('JsonItemExporterTest', 'test_nested_item', 477)": {'mod': [478, 479, 480, 485]}, "('JsonItemExporterTest', 'test_nested_dict_item', 488)": {'mod': [490]}, "('CustomItemExporterTest', None, 509)": {'mod': [509]}, "('CustomItemExporterTest', 'test_exporter_custom_serializer', 511)": {'mod': [519, 522, 523]}}}]}


Print and store how many tokens were used in memory/logs

In this way, we can also store this to benchmark results.

A huge increase in tokens will not be worth a minor improvement in benchmark resultss.


Push to Hub fails with `model_name`

### System Info

- `transformers` version: 4.25.0.dev0
- Platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.31
- Python version: 3.9.13
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from datasets import load_dataset, DatasetDict

common_voice = DatasetDict()

#common_voice["train"] = load_dataset("mozilla-foundation/common_voice_11_0", "sv-SE", split="train+validation", use_auth_token=True)
#common_voice["test"] = load_dataset("mozilla-foundation/common_voice_11_0", "sv-SE", split="test", use_auth_token=True)

common_voice["train"] = load_dataset("mozilla-foundation/common_voice_11_0", "sv-SE", split="train[:1%]+validation[:1%]", use_auth_token=True)
common_voice["test"] = load_dataset("mozilla-foundation/common_voice_11_0", "sv-SE", split="test[:1%]", use_auth_token=True)

print(common_voice)

common_voice = common_voice.remove_columns(["accent", "age", "client_id", "down_votes", "gender", "locale", "path", "segment", "up_votes"])

print(common_voice)

from transformers import WhisperFeatureExtractor

feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-small")

from transformers import WhisperTokenizer

tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-small", language="swedish", task="transcribe")

from transformers import WhisperProcessor

processor = WhisperProcessor.from_pretrained("openai/whisper-small", language="swedish", task="transcribe")

print(common_voice["train"][0])

from datasets import Audio

common_voice = common_voice.cast_column("audio", Audio(sampling_rate=16000))

print(common_voice["train"][0])

def prepare_dataset(batch):
    # load and resample audio data from 48 to 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array
    batch["input_features"] = feature_extractor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]

    # encode target text to label ids
    batch["labels"] = tokenizer(batch["sentence"]).input_ids
    return batch

common_voice = common_voice.map(prepare_dataset, remove_columns=common_voice.column_names["train"], num_proc=1)

import torch

from dataclasses import dataclass
from typing import Any, Dict, List, Union

@dataclass
class DataCollatorSpeechSeq2SeqWithPadding:
    processor: Any

    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        # split inputs and labels since they have to be of different lengths and need different padding methods
        # first treat the audio inputs by simply returning torch tensors
        input_features = [{"input_features": feature["input_features"]} for feature in features]
        batch = self.processor.feature_extractor.pad(input_features, return_tensors="pt")

        # get the tokenized label sequences
        label_features = [{"input_ids": feature["labels"]} for feature in features]
        # pad the labels to max length
        labels_batch = self.processor.tokenizer.pad(label_features, return_tensors="pt")

        # replace padding with -100 to ignore loss correctly
        labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)

        # if bos token is appended in previous tokenization step,
        # cut bos token here as it's append later anyways
        if (labels[:, 0] == self.processor.tokenizer.bos_token_id).all().cpu().item():
            labels = labels[:, 1:]

        batch["labels"] = labels

        return batch

"""Let's initialise the data collator we've just defined:"""

data_collator = DataCollatorSpeechSeq2SeqWithPadding(processor=processor)

import evaluate

metric = evaluate.load("wer")

def compute_metrics(pred):
    pred_ids = pred.predictions
    label_ids = pred.label_ids

    # replace -100 with the pad_token_id
    label_ids[label_ids == -100] = tokenizer.pad_token_id

    # we do not want to group tokens when computing the metrics
    pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
    label_str = tokenizer.batch_decode(label_ids, skip_special_tokens=True)

    wer = 100 * metric.compute(predictions=pred_str, references=label_str)

    return {"wer": wer}

from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")

model.config.forced_decoder_ids = None
model.config.suppress_tokens = []

from transformers import Seq2SeqTrainingArguments

training_args = Seq2SeqTrainingArguments(
    output_dir="./whisper-small-sv-test2",  # change to a repo name of your choice
    per_device_train_batch_size=16,
    gradient_accumulation_steps=1,  # increase by 2x for every 2x decrease in batch size
    learning_rate=1e-5,
    warmup_steps=500,
    max_steps=10,
    gradient_checkpointing=True,
    fp16=True,
    group_by_length=True,
    evaluation_strategy="steps",
    per_device_eval_batch_size=8,
    predict_with_generate=True,
    generation_max_length=225,
    save_steps=1000,
    eval_steps=1000,
    logging_steps=25,
    report_to=["tensorboard"],
    load_best_model_at_end=True,
    metric_for_best_model="wer",
    greater_is_better=False,
    push_to_hub=True,
)

from transformers import Seq2SeqTrainer

trainer = Seq2SeqTrainer(
    args=training_args,
    model=model,
    train_dataset=common_voice["train"],
    eval_dataset=common_voice["test"],
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    tokenizer=processor.feature_extractor,
)

trainer.train()

"""Our best WER is 32.0% - not bad for 8h of training data! We can submit our checkpoint to the [`hf-speech-bench`](https://huggingface.co/spaces/huggingface/hf-speech-bench) on push by setting the appropriate key-word arguments (kwargs):"""

kwargs = {
    "dataset_tags": "mozilla-foundation/common_voice_11_0",
    "dataset": "Common Voice 11.0",  # a 'pretty' name for the training dataset
    "language": "sv",
    #"model_name": "WhisperSmallSwedishBirgerMoell",  # a 'pretty' name for our model
    "finetuned_from": "openai/whisper-small",
    "tasks": "automatic-speech-recognition",
    "tags": "hf-asr-leaderboard",
}

trainer.push_to_hub(**kwargs)

from transformers import pipeline
import gradio as gr

pipe = pipeline(model="birgermoell/whisper-small-sv-test2")  # change to "your-username/the-name-you-picked"

def transcribe(audio):
    text = pipe(audio)["text"]
    return text

iface = gr.Interface(
    fn=transcribe,
    inputs=gr.Audio(source="microphone", type="filepath"),
    outputs="text",
    title="Whisper Small SV",
    description="Realtime demo for Swedish speech recognition using a fine-tuned Whisper small model.",
)

iface.launch()

```

### Expected behavior

The following script is a downloaded version of the colab notebook that follows the whisper fine-tuning tutorial.
https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/fine_tune_whisper.ipynb

One edit was that I removed the model name since I had an issue that it was complaining about two model names that made it impossible to upload. The script just runs on 1% of the dataset on 10 epochs.

kwargs = {
    "dataset_tags": "mozilla-foundation/common_voice_11_0",
    "dataset": "Common Voice 11.0",  # a 'pretty' name for the training dataset
    "language": "sv",
    #"model_name": "WhisperSmallSwedishBirgerMoell",  # a 'pretty' name for our model
    "finetuned_from": "openai/whisper-small",
    "tasks": "automatic-speech-recognition",
    "tags": "hf-asr-leaderboard",
}

https://huggingface.co/birgermoell/whisper-small-sv-test2

I also ran into similar issues when I trained a model on the whole dataset.

https://huggingface.co/birgermoell/whisper-small-sv

{'base_commit': '6dda14dc47d82f0e32df05fea8ba6444ba52b90a', 'files': [{'path': 'src/transformers/models/clip/processing_clip.py', 'status': 'modified', 'Loc': {"('CLIPProcessor', 'decode', 102)": {'add': [107]}}}, {'path': 'src/transformers/models/flava/processing_flava.py', 'status': 'modified', 'Loc': {"('FlavaProcessor', 'decode', 119)": {'add': [124]}}}, {'path': 'src/transformers/models/layoutlmv2/processing_layoutlmv2.py', 'status': 'modified', 'Loc': {"('LayoutLMv2Processor', 'decode', 155)": {'add': [160]}}}, {'path': 'src/transformers/models/layoutlmv3/processing_layoutlmv3.py', 'status': 'modified', 'Loc': {"('LayoutLMv3Processor', 'decode', 153)": {'add': [158]}}}, {'path': 'src/transformers/models/layoutxlm/processing_layoutxlm.py', 'status': 'modified', 'Loc': {"('LayoutXLMProcessor', 'decode', 155)": {'add': [160]}}}, {'path': 'src/transformers/models/markuplm/processing_markuplm.py', 'status': 'modified', 'Loc': {"('MarkupLMProcessor', 'decode', 135)": {'add': [140]}}}, {'path': 'src/transformers/models/owlvit/processing_owlvit.py', 'status': 'modified', 'Loc': {"('OwlViTProcessor', 'decode', 156)": {'add': [161]}}}, {'path': 'src/transformers/models/vilt/processing_vilt.py', 'status': 'modified', 'Loc': {"('ViltProcessor', 'decode', 103)": {'add': [108]}}}, {'path': 'src/transformers/models/vision_text_dual_encoder/processing_vision_text_dual_encoder.py', 'status': 'modified', 'Loc': {"('VisionTextDualEncoderProcessor', None, 25)": {'add': [129]}}}, {'path': 'src/transformers/models/x_clip/processing_x_clip.py', 'status': 'modified', 'Loc': {"('XCLIPProcessor', 'decode', 104)": {'add': [109]}}}, {'path': 'src/transformers/processing_utils.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [229]}}}, {'path': 'tests/models/clip/test_processor_clip.py', 'status': 'modified', 'Loc': {"('CLIPProcessorTest', 'test_tokenizer_decode', 178)": {'add': [189]}}}, {'path': 'tests/models/flava/test_processor_flava.py', 'status': 'modified', 'Loc': {"('FlavaProcessorTest', 'test_tokenizer_decode', 222)": {'add': [233]}}}, {'path': 'tests/models/layoutlmv2/test_processor_layoutlmv2.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [21]}, "('LayoutLMv2ProcessorTest', None, 37)": {'add': [88, 135]}}}, {'path': 'tests/models/layoutlmv3/test_processor_layoutlmv3.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [21, 148]}, "('LayoutLMv3ProcessorTest', None, 37)": {'add': [101]}}}, {'path': 'tests/models/layoutxlm/test_processor_layoutxlm.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [21]}, "('LayoutXLMProcessorTest', None, 43)": {'add': [76, 128]}}}, {'path': 'tests/models/markuplm/test_processor_markuplm.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [135]}}}, {'path': 'tests/models/mctct/test_processor_mctct.py', 'status': 'modified', 'Loc': {"('MCTCTProcessorTest', 'test_tokenizer_decode', 135)": {'add': [146]}}}, {'path': 'tests/models/owlvit/test_processor_owlvit.py', 'status': 'modified', 'Loc': {"('OwlViTProcessorTest', 'test_tokenizer_decode', 230)": {'add': [241]}}}, {'path': 'tests/models/speech_to_text/test_processor_speech_to_text.py', 'status': 'modified', 'Loc': {"('Speech2TextProcessorTest', 'test_tokenizer_decode', 135)": {'add': [146]}}}, {'path': 'tests/models/vision_text_dual_encoder/test_processor_vision_text_dual_encoder.py', 'status': 'modified', 'Loc': {"('VisionTextDualEncoderProcessorTest', 'test_tokenizer_decode', 159)": {'add': [170]}}}, {'path': 'tests/models/wav2vec2/test_processor_wav2vec2.py', 'status': 'modified', 'Loc': {"('Wav2Vec2ProcessorTest', 'test_tokenizer_decode', 128)": {'add': [139]}}}, {'path': 'tests/models/wav2vec2_with_lm/test_processor_wav2vec2_with_lm.py', 'status': 'modified', 'Loc': {"('Wav2Vec2ProcessorWithLMTest', None, 49)": {'add': [369]}}}, {'path': 'tests/models/whisper/test_processor_whisper.py', 'status': 'modified', 'Loc': {"('WhisperProcessorTest', 'test_tokenizer_decode', 107)": {'add': [118]}}}]}


networking
module
affects_2.4
support:community
aci
feature
cisco

aci_epg module needs to support PreferredGroup

##### ISSUE TYPE
 - Feature Idea

##### COMPONENT NAME
aci_epg

##### ANSIBLE VERSION
```
[root@ansible-server ~]# ansible --version
ansible 2.4.0.0
  config file = None
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
[root@ansible-server ~]#
```

##### CONFIGURATION

[root@ansible-server ~]# ansible-config dump --only-changed
[root@ansible-server ~]#

##### OS / ENVIRONMENT
Ansible server on CentOS 7.3 and ACI version 3.0 or 3.1

##### SUMMARY

Since ACI 2.3 it is possible to configure EPGs to be part of a Preferred Group. This is a new attribute of the fvAEPg object. EPGs that are part of the Preferred Group can communicate without contracts. This is very convenient for migration scenarios as well as customers that implement ACI for network automation but not for policy.

##### STEPS TO REPRODUCE

##### EXPECTED RESULTS

The module should have a new option to configure.

preferred_group: yes, no

The object to configure is:

fvAEPg.attributes.prefGrMemb and the option is "include" or "exclude".

##### ACTUAL RESULTS

```

```


enhancement
good first issue
frontend
large effort

Update the frontend to use i18n keys

**What problem or use case are you trying to solve?**
The new UI hardcodes english text throughout the app. In order to support i18n, we should extend our i18n provider and replaced the hardcoded values with the new keys

**Describe the UX of the solution you'd like**

**Do you have thoughts on the technical implementation?**

**Describe alternatives you've considered**

**Additional context**

{'base_commit': '4405b109e3abcd197666430708de2881e7cde8da', 'files': [{'path': 'frontend/src/components/form/custom-input.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0, 15], 'mod': [21]}}}, {'path': 'frontend/src/components/form/settings-form.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [7, 13, 17, 37], 'mod': [10, 12, 15, 164, 174, 193, 223, 237, 244, 258, 294, 337, 348, 352, 359, 372, 373, 376, 380, 390, 391, 393, 395]}}}, {'path': 'frontend/src/components/modals/AccountSettingsModal.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [2, 11, 25], 'mod': [89, 95, 125, 129]}}}, {'path': 'frontend/src/components/modals/ConnectToGitHubByTokenModal.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1, 9, 13], 'mod': [32, 33, 38]}}}, {'path': 'frontend/src/components/modals/LoadingProject.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0, 3, 30], 'mod': [34]}}}, {'path': 'frontend/src/components/modals/connect-to-github-modal.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1, 10, 18], 'mod': [27, 34, 58, 64]}}}, {'path': 'frontend/src/components/modals/security/Security.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1, 3, 19], 'mod': [24]}}}, {'path': 'frontend/src/components/modals/security/invariant/Invariant.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [126, 137, 146, 165, 167, 198, 200, 217, 219, 224, 267, 281, 284, 285, 292, 301, 307, 313]}}}, {'path': 'frontend/src/components/project-menu/project-menu-details-placeholder.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0, 2, 12], 'mod': [15]}}}, {'path': 'frontend/src/components/project-menu/project-menu-details.tsx', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0, 2, 14], 'mod': [35]}}}, {'path': 'frontend/src/i18n/translation.json', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1519], 'mod': [801]}}}]}


[RFC?] Make cross_val_score output a dict/named tuple.

Two major things here -
- Often I see that only a partial output of `_fit_and_score` is taken for use. It is wasteful to generate and discard arrays. It would rather be much better to generate only the stuff that is required.
- Now that we have more options, like @jnothman says [here](https://github.com/scikit-learn/scikit-learn/pull/7325#issuecomment-246529168) and [here](https://github.com/scikit-learn/scikit-learn/pull/7388#issuecomment-246233650) should we modify the output of `cross_val_score` (and also `_fit_and_score` to be a dict or a named tuple similar to the structure of `cv_results_`? (I think named-tuple is a better choice atleast for `_fit_and_score` as we stack the result of multiple `_fit_and_score` operations via `Parallel` mostly)

If we are changing the output of `cross_val_score`, this would be an ideal time to do it as we don't have to deprecate anything...

@jnothman @amueller @vene @GaelVaroquaux @agramfort


[Trainer] Why not use `tqdm`'s `dynamic_ncols=True` option?

### Feature request

# Problem

Tqdm progress bar is getting ugly when the width of the terminal is shrunk!

![image](https://github.com/huggingface/transformers/assets/4879345/b60f232f-41a5-40de-b759-8bb2710d3b5f)

It progress bar makes the new line on every update! It is very ugly...

# Solution

Simply add the `dynamic_ncols=True` option to `tqdm`. It is located in `trainer_callbacks.ProgressCallback`.

![image](https://github.com/huggingface/transformers/assets/4879345/6741eb00-7430-48db-acc8-4c3a0eb00217)

You can check the progress bar is now dynamically resized when the terminal size is updated.

### Motivation

When I connect `tmux` session with different widths of the terminal, then the `tqdm` printing is getting ugly.

### Your contribution

Please check the PR #24101

{'base_commit': 'a73883ae9ec66cb35a8222f204a5f2fafc326d3f', 'files': [{'path': 'src/transformers/trainer_callback.py', 'status': 'modified', 'Loc': {"('ProgressCallback', 'on_train_begin', 474)": {'mod': [476]}, "('ProgressCallback', 'on_prediction_step', 484)": {'mod': [487]}}}]}


Shell alias clobbers some history lines

I was trying this out, and found that large swathes of my history were missing after running "fuck" a single time. This should _not_ modify history except to insert a command it executes..


I have this simple spider, when I call `scrapy crawl dataspider` it works fine and prints the item in the  output :

    import json
    from scrapy.spiders import Spider

    class dataspider(Spider):
        name='dataspider'
        start_urls=('https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL',)
        def parse(self, response):
            j=json.loads( response.body.decode('utf-8') )
            yield j['matches'][1]

Outputs :

> {'t': 'AAPL', 'n': 'Apple Inc.', 'e': 'NASDAQ', 'id': '22144'}

However as soon as I try to save the item in a file using `scrapy crawl dataspider -o out.json` I get this error :

> AttributeError: 'FeedExporter' object has no attribute 'slot'

Full Traceback is :

```
$ scrapy crawl dataspider -o ./test.json
2017-01-30 14:32:06 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: googlefinance)
2017-01-30 14:32:06 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'googlefinance', 'CONCURRENT_REQUESTS': 100, 'CONCURRENT_REQUESTS_PER_DOMAIN': 100, 'DNS_TIMEOUT': 30, 'DOWNLOAD_TIMEOUT': 30, 'FEED_FORMAT': 'json', 'FEED_URI': './test.json', 'NEWSPIDER_MODULE': 'googlefinance.spiders', 'RETRY_HTTP_CODES': [500, 502, 503, 504, 400, 403, 404, 408], 'RETRY_TIMES': 30, 'SPIDER_MODULES': ['googlefinance.spiders'], 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FSL 7.0.6.01001)'}
2017-01-30 14:32:06 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2017-01-30 14:32:06 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-01-30 14:32:06 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-01-30 14:32:06 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-01-30 14:32:06 [scrapy.core.engine] INFO: Spider opened
2017-01-30 14:32:06 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.open_spider of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 187, in open_spider
    uri = self.urifmt % self._get_uri_params(spider)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 262, in _get_uri_params
    params[k] = getattr(spider, k)
  File "/usr/lib/python3.6/site-packages/scrapy/spiders/__init__.py", line 36, in logger
    logger = logging.getLogger(self.name)
  File "/usr/lib/python3.6/logging/__init__.py", line 1813, in getLogger
    return Logger.manager.getLogger(name)
  File "/usr/lib/python3.6/logging/__init__.py", line 1167, in getLogger
    raise TypeError('A logger name must be a string')
TypeError: A logger name must be a string
2017-01-30 14:32:06 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-30 14:32:06 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-01-30 14:32:07 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL> (referer: None)
2017-01-30 14:32:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL>
{'t': 'AAPL', 'n': 'Apple Inc.', 'e': 'NASDAQ', 'id': '22144'}
2017-01-30 14:32:07 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.item_scraped of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 217, in item_scraped
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
2017-01-30 14:32:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL>
{'t': 'AAPL', 'n': 'APPLE INC CEDEAR(REPR 1/10 SHR)', 'e': 'BCBA', 'id': '640373807586235'}
2017-01-30 14:32:07 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.item_scraped of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 217, in item_scraped
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
2017-01-30 14:32:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL>
{'t': 'AAPL', 'n': 'Apple', 'e': 'SWX', 'id': '268194557752272'}
2017-01-30 14:32:07 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.item_scraped of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 217, in item_scraped
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
2017-01-30 14:32:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL>
{'t': 'AVSPY', 'n': 'NASDAQ OMX Alpha AAPL vs. SPY Index', 'e': 'INDEXNASDAQ', 'id': '3139928'}
2017-01-30 14:32:07 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.item_scraped of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 217, in item_scraped
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
2017-01-30 14:32:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL>
{'t': 'AAPL34', 'n': 'APPLE DRN', 'e': 'BVMF', 'id': '486420404817650'}
2017-01-30 14:32:07 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.item_scraped of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 217, in item_scraped
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
2017-01-30 14:32:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL>
{'t': 'AAPL', 'n': 'APPLE COMPUTER INC', 'e': 'BMV', 'id': '119565461895124'}
2017-01-30 14:32:07 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.item_scraped of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 217, in item_scraped
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
2017-01-30 14:32:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL>
{'t': 'AAPL-EUR', 'n': 'Apple', 'e': 'SWX', 'id': '706336206708362'}
2017-01-30 14:32:07 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.item_scraped of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 217, in item_scraped
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
2017-01-30 14:32:07 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com/finance/match?matchtype=matchall&ei=UVlPWNmDEYm_U7SqgvAH&q=AAPL>
{'t': 'AAPL-USD', 'n': 'Apple', 'e': 'SWX', 'id': '1009743014824088'}
2017-01-30 14:32:07 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.item_scraped of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 217, in item_scraped
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
2017-01-30 14:32:07 [scrapy.core.engine] INFO: Closing spider (finished)
2017-01-30 14:32:07 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method FeedExporter.close_spider of <scrapy.extensions.feedexport.FeedExporter object at 0x7ff68de97ef0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/lib/python3.6/site-packages/scrapy/extensions/feedexport.py", line 198, in close_spider
    slot = self.slot
AttributeError: 'FeedExporter' object has no attribute 'slot'
2017-01-30 14:32:07 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 309,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 761,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 1, 30, 13, 32, 7, 192220),
 'item_scraped_count': 8,
 'log_count/DEBUG': 10,
 'log_count/ERROR': 10,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2017, 1, 30, 13, 32, 6, 846350)}
2017-01-30 14:32:07 [scrapy.core.engine] INFO: Spider closed (finished)))

```

Any idea what the problem is ?

{'base_commit': 'bd8c293a97f7f08989cff1db0d9c32f5a2208b77', 'files': [{'path': 'scrapy/spiderloader.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [2]}, "('SpiderLoader', '_load_all_spiders', 26)": {'mod': [28, 29]}}}, {'path': 'tests/test_spiderloader/__init__.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [3]}, "('SpiderLoaderTest', 'test_crawler_runner_loading', 82)": {'add': [91]}}}]}


NumPy dev causes test errors due to use of np.matrix

We are getting many warnings like `PendingDeprecationWarning('the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.` using numpy master (see logs at https://travis-ci.org/scikit-learn/scikit-learn/builds/387352026)

Apart from a very long log, this causes test failures where we have used `assert_no_warnings` (which we could now be importing from numpy instead of having our own implementation).

It might be a good idea to remove all uses of np.matrix that raise warnings. On the other hand, we might also consider that `assert_no_warnings` shouldn't be bothered by `PendingDeprecationWarning`s.

{'base_commit': 'bb385394b87e382a34db829bc7ed60d347af73c9', 'files': [{'path': 'sklearn/ensemble/tests/test_iforest.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [8], 'mod': [18]}, "(None, 'test_iforest_error', 91)": {'mod': [108, 109]}}}]}


good first issue
site-bug
patch-available

[cbc.ca] "unable to extract OpenGraph description"

### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

United States

### Provide a description that is worded well enough to be understood

Video from CBC's site will not download, throws an error saying "unable to extract OpenGraph description", then says it's finished downloading the playlist (but downloaded no video files).

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', 'https://www.cbc.ca/player/play/video/1.3594815']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.04.28.232723 from yt-dlp/yt-dlp-nightly-builds [ac817bc83] (pip)
[debug] Python 3.10.11 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 2023-03-02-git-814178f926-full_build-www.gyan.dev (setts), ffprobe 2023-03-02-git-814178f926-full_build-www.gyan.dev, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.06.15, mutagen-1.46.0, requests-2.31.0, sqlite3-3.40.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1810 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.04.28.232723 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.04.28.232723 from yt-dlp/yt-dlp-nightly-builds)
[generic] Extracting URL: https://www.cbc.ca/player/play/video/1.3594815
[generic] 1: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] 1: Extracting information
[debug] Looking for embeds
[debug] Identified a twitter:player iframe
[cbc.ca] Extracting URL: https://www.cbc.ca/i/phoenix/player/syndicate/?autoPlay=true&sourceId=1.3594815
[cbc.ca] syndicate: Downloading webpage
WARNING: [cbc.ca] unable to extract OpenGraph description; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[download] Downloading playlist: CBC Player
[cbc.ca] Playlist CBC Player: Downloading 0 items
[download] Finished downloading playlist: CBC Player
```


/privateGPT/gpt4all_j.py", line 152, in _call
    text = self.client.generate(
TypeError: generate() got an unexpected keyword argument 'new_text_callback'

{'base_commit': '77447e50c0b8143edcf34896af80dd58925582f9', 'files': [{'path': 'requirements.txt', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [1]}}}]}


Add xlsx Export

<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->

## Checklist
<!--
Put x into all boxes (like this [x]) once you have completed what they say.
Make sure complete everything in the checklist.
-->
- [x] I'm reporting a feature request
- [x] I've checked for similar feature requests including closed ones

## Description
<!--
Provide a detailed description of the feature you would like Sherlock to have
-->

WRITE DESCRIPTION HERE

Add an option to export the result on xlsx file type.

{'base_commit': '9db8c213ffdad873380c9de41c142923ba0dc260', 'files': [{'path': '.gitignore', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [24]}}}, {'path': 'requirements.txt', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [7]}}}, {'path': 'sherlock/sherlock.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [10]}, "(None, 'main', 477)": {'add': [508, 718]}}}]}


ollama llm vision api call error:     async for raw_chunk in stream_resp: TypeError: 'async for' requires an object with __aiter__ method, got bytes

**Bug description**
when running any vision llm call (like example/llm_vision.py)
there seems to be an issue with async def _achat_completion_stream(self, messages: list[dict], timeout: int = USE_CONFIG_TIMEOUT) -> str: method

**Bug solved method**
No solve yet

**Environment information**
system metal, llm ollama, Python 3.10.13

- LLM type and model name: ollama, llava latest
- System version:
- Python version: Python 3.10.13
- MetaGPT version or branch: main

- packages version: /
- installation method: from source

**Screenshots or logs**
.....
    do = self.iter(retry_state=retry_state)
    return fut.result()
    return self.__get_result()
    raise self._exception
    result = await fn(*args, **kwargs)
    return await self._achat_completion_stream(messages, timeout=self.get_timeout(timeout))
    async for raw_chunk in stream_resp: (HERE)
TypeError: 'async for' requires an object with __aiter__ method, got bytes


Example_scenes.py run problem question

I was able to get the install success thanks to help.  I ran the example_scenes.py file and have the results below.  I am now also going  through https://talkingphysics.wordpress.com/2019/01/08/getting-started-animating-with-manim-and-python-3-7/ and have similar errors when running the first run python -m manim pymanim_tutorial_P37.py Shapes -pl.  So I am trying to crawl before walking and would like to get through example_scenes and first tutorial .py run with success so any help is appreciated.

C:\Users\Admin\Desktop\manim-master>python ./manim.py example_scenes.py SquareTo
Circle -pl
Media will be written to ./media\. You can change this behavior with the --media
_dir flag.
[concat @ 0000000000375a40] Impossible to open 'CC:/Users/Admin/Desktop/manim-ma
ster/media/videos/example_scenes/480p15/partial_movie_files/SquareToCircle/00000
.mp4'
C:\Users\Admin\Desktop\manim-master\media\videos\example_scenes\480p15\partial_m
ovie_files\SquareToCircle\partial_movie_file_list.txt: Protocol not found
Did you mean file:C:\Users\Admin\Desktop\manim-master\media\videos\example_scene
s\480p15\partial_movie_files\SquareToCircle\partial_movie_file_list.txt?

File ready at C:\Users\Admin\Desktop\manim-master\media\videos\example_scenes\48
0p15\SquareToCircle.mp4

Played 3 animations

Traceback (most recent call last):
  File "C:\Users\Admin\Desktop\manim-master\manimlib\extract_scene.py", line 156
, in main
    open_file_if_needed(scene.file_writer, **config)
  File "C:\Users\Admin\Desktop\manim-master\manimlib\extract_scene.py", line 35,
 in open_file_if_needed
    os.startfile(file_path)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\\
Users\\Admin\\Desktop\\manim-master\\media\\videos\\example_scenes\\480p15\\Squa
reToCircle.mp4'

{'base_commit': 'dbdd7996960ba46ed044a773290b02f17478c760', 'files': [{'path': 'manimlib/scene/scene_file_writer.py', 'status': 'modified', 'Loc': {"('SceneFileWriter', 'combine_movie_files', 253)": {'mod': [289]}}}]}


[youtube] YouTube serving erroneous DASH Manifest VP9 formats

<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->

## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.07. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running yt-dlp version **2021.07.07**
- [x] I've checked that all provided URLs are alive and playable in a browser (but with condition, see below)
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones

## Verbose log

```
$ yt-dlp V_h3Z40AAtw -F
[youtube] V_h3Z40AAtw: Downloading webpage
[youtube] V_h3Z40AAtw: Downloading MPD manifest
[info] Available formats for V_h3Z40AAtw:
ID  EXT  RESOLUTION FPS |  FILESIZE   TBR PROTO | VCODEC        VBR ACODEC     ABR  ASR    NOTE
--- ---- ---------- --- - --------- ----- ----- - ----------- ----- --------- ---- ------- ---------------------------------------
139 m4a  audio only     |  1.52MiB    50k dash  |                   mp4a.40.5  50k 22050Hz DASH audio, m4a_dash, 22050Hz
140 m4a  audio only     |  4.02MiB   129k https |                   mp4a.40.2 129k 44100Hz audio_quality_medium, m4a_dash, 44100Hz
160 mp4  256x144    30  |            108k dash  | avc1.4d400b  108k                        DASH video, mp4_dash
278 webm 256x144    30  |             95k dash  | vp9           95k                        DASH video, webm_dash
133 mp4  426x240    30  |            242k dash  | avc1.4d400c  242k                        DASH video, mp4_dash
242 webm 426x240    30  |            220k dash  | vp9          220k                        DASH video, webm_dash
134 mp4  640x360    30  |  19.25MiB  620k https | avc1.4d401e  620k                        360p, mp4_dash
18  mp4  640x360    30  |  22.68MiB  730k https | avc1.42001E  730k mp4a.40.2   0k 44100Hz 360p, 44100Hz
243 webm 640x360    30  |            405k dash  | vp9          405k                        DASH video, webm_dash
135 mp4  854x480    30  |           1155k dash  | avc1.4d400c 1155k                        DASH video, mp4_dash
244 webm 854x480    30  |            752k dash  | vp9          752k                        DASH video, webm_dash
136 mp4  1280x720   30  |  69.87MiB 2251k https | avc1.4d401f 2251k                        720p, mp4_dash
22  mp4  1280x720   30  |           2380k https | avc1.64001F 2380k mp4a.40.2   0k 44100Hz 720p, 44100Hz
247 webm 1280x720   30  |           1505k dash  | vp9         1505k                        DASH video, webm_dash
248 webm 1920x1080  30  |           2646k dash  | vp9         2646k                        DASH video, webm_dash

$ yt-dlp -v V_h3Z40AAtw
[debug] Command-line config: ['-v', 'V_h3Z40AAtw']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] yt-dlp version 2021.07.13.1626134551 (zip)
[debug] Python version 3.9.6 (CPython 64bit) - Linux-5.8.0-41-generic-x86_64-with-glibc2.32
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: V_h3Z40AAtw
[youtube] V_h3Z40AAtw: Downloading webpage
[youtube] [debug] Fetching webpage from https://www.youtube.com/watch?v=V_h3Z40AAtw&bpctr=9999999999&has_verified=1
[youtube] V_h3Z40AAtw: Downloading MPD manifest
[youtube] [debug] Fetching webpage from https://manifest.googlevideo.com/api/manifest/dash/expire/1626251414/ei/NkzuYLz-Ap-0s8IPlf2YyAo/ip/2001%3A19f0%3A7001%3A13a1%3A5400%3A3ff%3Afe11%3A205f/id/57f877678d0002dc/source/youtube/requiressl/yes/playback_host/r2---sn-oguelne7.googlevideo.com/mh/t1/mm/31%2C29/mn/sn-oguelne7%2Csn-oguesnzz/ms/au%2Crdu/mv/m/mvi/2/pl/55/tx/24027688/txs/24027687%2C24027688%2C24027689%2C24027690/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1267500/vprv/1/mt/1626229496/fvip/2/keepalive/yes/fexp/24001373%2C24007246/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Ctx%2Ctxs%2Chfr%2Cas%2Cvprv%2Citag/sig/AOq0QJ8wRAIgel_8rJx7O1ChqaQTDiBI5cysHmZ_4uCmgCWN_kPxy8cCIDfgAFVDYl5WO7a1gLifSDw6vBfjELblxSgkOodOm1am/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/AG3C_xAwRAIgJVqT1xhW2KIhXVIj6cJRomDQ7-UOvq8yyC_J5r7ksfMCIBG0VgIJSIlNe49rl6ty6WA_DuH2AhKJvLOpq8fUBojv
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] V_h3Z40AAtw: Downloading 1 format(s): 248+140
[debug] locking youtube_V_h3Z40AAtw.lock
[debug] Invoking downloader on "https://manifest.googlevideo.com/api/manifest/dash/expire/1626251414/ei/NkzuYLz-Ap-0s8IPlf2YyAo/ip/2001%3A19f0%3A7001%3A13a1%3A5400%3A3ff%3Afe11%3A205f/id/57f877678d0002dc/source/youtube/requiressl/yes/playback_host/r2---sn-oguelne7.googlevideo.com/mh/t1/mm/31%2C29/mn/sn-oguelne7%2Csn-oguesnzz/ms/au%2Crdu/mv/m/mvi/2/pl/55/tx/24027688/txs/24027687%2C24027688%2C24027689%2C24027690/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1267500/vprv/1/mt/1626229496/fvip/2/keepalive/yes/fexp/24001373%2C24007246/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Ctx%2Ctxs%2Chfr%2Cas%2Cvprv%2Citag/sig/AOq0QJ8wRAIgel_8rJx7O1ChqaQTDiBI5cysHmZ_4uCmgCWN_kPxy8cCIDfgAFVDYl5WO7a1gLifSDw6vBfjELblxSgkOodOm1am/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/AG3C_xAwRAIgJVqT1xhW2KIhXVIj6cJRomDQ7-UOvq8yyC_J5r7ksfMCIBG0VgIJSIlNe49rl6ty6WA_DuH2AhKJvLOpq8fUBojv"
[dashsegments] Total fragments: 50
[download] Destination: Sweet Candy ②-V_h3Z40AAtw.f248.webm
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 1 of 10) ...
^C[debug] unlocking youtube_V_h3Z40AAtw.lock

ERROR: Interrupted by user
```
<!--
Do not remove the above ```
-->

## Description
[link to video](https://youtu.be/V_h3Z40AAtw)

The video itself plays on browser, and doesn't have 1080p as you can see it.

But yt-dlp (and youtube-dl) reports 1080p format, which possibly doesn't exist on the server. (format `248` on the video fails to download all segments.)

Resolutions shown in webpage here:
![image](https://user-images.githubusercontent.com/10355528/125551541-4e14ac3a-0f66-40e2-9931-0a4f25e04750.png)

__Edit:__ Tested web and android clients, some locations (JP, Vultr JP, OCI US?), with cookies or not, but all of them has this "ghosty" format


Face on angles >180 degrees not recognized on extraction

Hi guys thanks for the amazing work here!

I have been following the landmark detection dialogue  #187 and have tried both hog and cnn with both face-alignment and face_recognition. I got face-alignment with cnn working great, with pytorch on win10 now. However, I noticed that none of the above are able to reliably able to identify faces where the face is pointing downwards, for example with the forehead pointing from 6 to 9 o'clock.

I think all these algorithms tend to look for eyes being above the level of the mouth.

For example [Image Removed) this image would not be detected and extracted by hog or cnn in face-alignment or face_recognition.

However by rotating it 90 deg to the right, so that the forehead is pointing up, makes it extracted.

Would it be possible to have an argument set to resend the image for alignment but rotated if it was not caught the first time?

I am ok with python and novice with git but could maybe even give it a try if someone points me to where the frame is passed for extraction.

Thanks!

{'base_commit': '59ca996eb1b510cef7ae60a179c36ea7f353f71e', 'files': [{'path': 'lib/cli.py', 'status': 'modified', 'Loc': {"('DirectoryProcessor', 'get_faces_alignments', 140)": {'add': [144]}, '(None, None, None)': {'mod': [9]}, "('DirectoryProcessor', None, 29)": {'mod': [157]}, "('DirectoryProcessor', 'get_faces', 157)": {'mod': [159]}}}, {'path': 'lib/faces_detect.py', 'status': 'modified', 'Loc': {"('DetectedFace', '__init__', 10)": {'add': [11]}, "(None, 'detect_faces', 3)": {'mod': [3, 7]}, "('DetectedFace', None, 9)": {'mod': [10]}}}, {'path': 'lib/utils.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [0, 32]}}}, {'path': 'scripts/convert.py', 'status': 'modified', 'Loc': {'(None, None, None)': {'mod': [9]}, "('ConvertImage', 'convert', 216)": {'mod': [229, 230]}}}, {'path': 'scripts/extract.py', 'status': 'modified', 'Loc': {"('ExtractTrainingData', 'add_optional_arguments', 22)": {'add': [68]}, "('ExtractTrainingData', None, 12)": {'add': [100]}, "('ExtractTrainingData', 'handleImage', 101)": {'add': [104, 117], 'mod': [102, 106, 107]}, '(None, None, None)': {'mod': [7]}}}]}


EOL Error when training

After pulling the latest commit today I am now getting the below error when trying to train.

**Command**
python faceswap.py train -A "D:\Fakes\Data\Dataset_A\Faces" -B "D:\Fakes\Data\Dataset_B\Faces" -m "D:\Fakes\Model" -p -s 100 -bs 80 -t LowMem

**Error**
Traceback (most recent call last):
  File "faceswap.py", line 12, in <module>
    from scripts.convert import ConvertImage
  File "D:\Fakes\faceswap\scripts\convert.py", line 100
    help="Erosion kernel size. (Masked converter only). Positive values apply erosion which reduces the edge \
                                                                                ^
SyntaxError: EOL while scanning string literal


Issue with 'char_to_token()' function of DistilBertTokenizerFast

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.1
- Platform: Google Colab
- Python version: 3.8
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.4.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: NA

### Who can help:   **tokenizers: @mfuntowicz**

## Information

Model I am using DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased') to tokenize Squad 2.0 train and validate dataset.

The problem arises when using below code snippet to add_token_positions (start and end position) as below from https://huggingface.co/transformers/custom_datasets.html:

_def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []
    for i in range(len(answers)):
        start_positions.append(**encodings.char_to_token(i, answers[i]['answer_start'])**)
        end_positions.append(**encodings.char_to_token(i, answers[i]['answer_end'] - 1**))
        # if None, the answer passage has been truncated
        if start_positions[-1] is None:
            start_positions[-1] = tokenizer.model_max_length
        if end_positions[-1] is None:
            end_positions[-1] = tokenizer.model_max_length
    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})

add_token_positions(train_encodings, train_answers)
add_token_positions(val_encodings, val_answers)_

The tasks I am working on is:
*Training model on SQUaD 2.0 using code given on https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0

## To reproduce

Steps to reproduce the behavior:

1. Follow the steps given on https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0 and then verify start and end position outcome using below code snippet in Expected behavior

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior:
- Start and End position are being defined using above code snippet which will be provided as training/validation data to model but end position is not derived as correct value due to some issue with char_to_token() function which is used to find out end position.
- Please find below snippet for verification that answer using start and end position after tokenization is not matching with actual answer.
- So the training data which is being fed to model after tokenization is incorrect

idx=8
print(f'Actual context: {train_contexts[idx]}')
print(f'Actual question: {train_questions[idx]}')
print(f"Actual answer: {train_answers[idx]['text']}")

start_position=train_encodings.char_to_token(idx,train_answers[idx]['answer_start'])
end_position =train_encodings.char_to_token(idx,train_answers[idx]['answer_end'])
print(f"Answer after tokenization: {tokenizer.convert_ids_to_tokens(train_encodings['input_ids'][idx][start_position:end_position])}")

OUTPUT:
**Actual context:** Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles "Crazy in Love" and "Baby Boy".
**Actual question:** When did Beyoncé rise to fame?
**Actual answer:** late 1990s
**Answer after tokenization:** ['late', '1990s', 'as', 'lead', 'singer', 'of', 'r', '&', 'b', 'girl', '-', 'group', 'destiny', "'", 's', 'child', '.', 'managed', 'by', 'her', 'father', ',', 'mathew', 'knowles', ',', 'the', 'group', 'became', 'one', 'of', 'the', 'world', "'", 's', 'best', '-', 'selling', 'girl', 'groups', 'of', 'all', 'time', '.', 'their', 'hiatus', 'saw', 'the', 'release', 'of', 'beyonce', "'", 's', 'debut', 'album', ',', 'dangerously', 'in', 'love', '(', '2003', ')', ',', 'which', 'established', 'her', 'as', 'a', 'solo', 'artist', 'worldwide', ',', 'earned', 'five', 'grammy', 'awards', 'and', 'featured', 'the', 'billboard', 'hot', '100', 'number', '-', 'one', 'singles', '"', 'crazy', 'in', 'love', '"', 'and', '"', 'baby', 'boy', '"', '.', '[SEP]', 'when', 'did', 'beyonce', 'rise', 'to', 'fame', '?', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']

{'base_commit': '5f7a07c0c867abedbb3ebf135915eeee56add24b', 'files': [{'path': 'docs/source/custom_datasets.rst', 'status': 'modified', 'Loc': {'(None, None, None)': {'add': [564], 'mod': [561, 562, 566]}}}]}