Training HuggingFace tokenizer - ignore_merges

[ykoyfman]

Looking through Llama3 changes, I see that "ignore_merges" was added as a property to support conversion from tiktoken models. Can a native HF tokenizer be trained using this property? It's not clear if this is possible with, say, train_new_from_iterator. CC @ArthurZucker - Thanks

[ArthurZucker]

I think that it's not the case yet, but we should support it!

[ArthurZucker]

(Anyone feel free to open a PR if you have time!)