Update model card
Browse files
README.md
CHANGED
|
@@ -43,6 +43,7 @@ Without the repo: tokenize the pair `(text, statement)`, run `AutoModelForSequen
|
|
| 43 |
## Limits
|
| 44 |
|
| 45 |
- It still makes mistakes: about one in eight answers on judgment questions is wrong. Example: for a production `DELETE` that "no human has reviewed", it correctly says the action is not safe to run without a human, but answers "no" to "A human should approve this action before it runs". Do not use it as the only safety check.
|
|
|
|
| 46 |
- It only outputs P(yes): every answer is yes or no (or "not sure" if you set abstention thresholds). It cannot refuse a question that is not yes/no.
|
| 47 |
- A "not sure" band with a guaranteed error rate has to be fitted on a few hundred labeled examples of your own inputs (`maya.conformal.Abstainer`).
|
| 48 |
- English only; 435M parameters, about 0.6 s per question on a laptop CPU (4 threads).
|
|
|
|
| 43 |
## Limits
|
| 44 |
|
| 45 |
- It still makes mistakes: about one in eight answers on judgment questions is wrong. Example: for a production `DELETE` that "no human has reviewed", it correctly says the action is not safe to run without a human, but answers "no" to "A human should approve this action before it runs". Do not use it as the only safety check.
|
| 46 |
+
- Code changes are a known weak spot: it says an unreviewed pull request that drops a database column can be merged. Maya v0.1 got that right.
|
| 47 |
- It only outputs P(yes): every answer is yes or no (or "not sure" if you set abstention thresholds). It cannot refuse a question that is not yes/no.
|
| 48 |
- A "not sure" band with a guaranteed error rate has to be fitted on a few hundred labeled examples of your own inputs (`maya.conformal.Abstainer`).
|
| 49 |
- English only; 435M parameters, about 0.6 s per question on a laptop CPU (4 threads).
|