It absolutely refuses to leave javascript alone.
I instructed it to not use javascript in html code. And then i modified the prompt up to the point where i told it several different ways to not use js, javascript or < script > tags. ( Guess what, huggingface does not sanitize it's inputs lol, i had to edit this comment because it totally ate the script tag )
It does not care, crams js everywhere, and has the audacity to tell me the code does not use javascript but won't work without it :)
I'm not sure what to think about it because it seems to produce working code, but it's terrible at following instructions.
I did more testing and this hopeless.
I literally gave it a full working example of how to do the thing I want without javascript, and it said it understands, and then proceeded to completely disregard the example or the prompt and smeared javascript all over the code.
Example piece of it's output:
</script>
</body>
</html>
In this response, I have provided a CSS-only solution
Thank you for this test. I've seen cases where it doesn't understand some instructions directly and fails to follow them. I'm currently occupied with 13B model dataset clean up and have 16k and 32k context lined up.
Improving it's instruction following ability will be one of the key things I'll aim. I'll learn whether it can work with fine tune or may require me to implement RLHF/RLAIF in some capacity.
Folks like you help me identify such cases and thanks for that.
If you want some testing cases, this is a link that describes a pure CSS use case for collapsible elements.
https://www.digitalocean.com/community/tutorials/css-collapsible
My prompt, more or less, consisted of copying the first example from that link, and telling to make me something similar but in different arrangement.
Later I even included an explaination describing use of invisible checkbox and :checked + sibling selector in CSS but it didn't care.
I think this will be a good test case, since it not even once went the right way :)