[all-roberta-large-v1]Embedding difference between DJL Java code and SententenceTransformer Python code when input text is large

[legalizenet]

I have following Python code that generates embedding and prints

model = SentenceTransformer('sentence-transformers/all-roberta-large-v1')
embedding = model.encode(sentence)
print (*embedding[0])

and following DJL Java code does the same

Criteria<String, float[]> criteria =
            Criteria.builder()
                    .setTypes(String.class, float[].class)
                    .optModelUrls(<local traced all-roberta-large-v1 model path>)
                    .optEngine("PyTorch")
                    .optArgument("normalize", "true")
                    .optTranslatorFactory(new TextEmbeddingTranslatorFactory())
                    .optProgress(new ProgressBar())
                    .build();
float[] embedding =  criteria.loadModel().newPredictor().predict(sentence);
System.out.println(Arrays.toString(embedding));

• When sentence is small e.g. "Hello World" ; both Python and Java code has identical embedding output
• When sentence is large ; python embedding output differs from Java embedding output. Why ?. anything To do with the max token size ?, if yes how to make fix it.

e.g. here is the large text that produced different embedding output between Python and Java code

"The person Java Developers report to often depends on the size and type of organization they work in. Those who work in an IT agency setting may report to a Senior Java Developer who oversees the entire team of Java Developers and reports on their performance and customer satisfaction metrics. If they work in a larger office setting, they may report to the IT manager within their department and will ask them any questions they have about the project or for any advice on complex development challenges. Some Java Developers work in smaller offices, where they report directly to the leadership team to determine the company’s organizational goals to help them create an effective application system.A good Java Developer should have extensive experience and knowledge in software development and working with web applications. They should have strong problem solving and critical thinking abilities to help them locate potential issues with the application programs and provide logical and quick solutions. Good Java Developers also have basic design and user-experience knowledge to build visually-appealing applications that are simple and comprehensive for users to understand and navigate. While they both share similar responsibilities of designing and updating applications and software systems, Full Stack and Java Developers both have key differences in their roles. Java Developers are computer programmers who specialize in building applications using primarily Java coding. Full Stack Developers are proficient in multiple coding languages and use both front and back end coding to design different websites and applications. They often work primarily on designing and implementing websites and may collaborate closely with graphic designers and web developers to create efficient, user-friendly sites that meet the clients needs and preferences.The core experience requirements for a Java Developer include experience with user interface design, statistical analysis and database structures. They should have experience working with a company where they have demonstrated the ability to write well-designed code that withstands testing and runs efficiently, as well as the ability to troubleshoot and debug code before it is released.Java Developers can have a wide range of education and work experience in coding. Most Java Developers have a bachelor’s or master’s degree in computer engineering or computer science. Some have also earned a certificate from Oracle, the creator of Java. People develop their Java programming skills through various methods, and many can demonstrate their skills by creating quality code along with providing the requisite documentation. Some of the common educational paths and training opportunities for a Java Developer include. Java Developers typically work in a company’s IT department building and adding updates to Java-based programs and applications. They typically analyze user requirements to properly implement various design updates and fixes according to users’ needs. Java Developers also collaborate closely with clients or leadership teams to determine the application objectives, business goals and functionality needs to help them properly build a strong Java-based application."

[frankfliu]

@legalizenet
The difference is truncate length. In DJL, we use tokenizer's default max_length(in this case is 128), while the python code use 256.

You can get the same result with the following code:

            Criteria<String, float[]> criteria =
                    Criteria.builder()
                            .setTypes(String.class, float[].class)
                            .optModelUrls("djl://ai.djl.huggingface.pytorch/sentence-transformers/all-roberta-large-v1")
                            .optTranslatorFactory(new TextEmbeddingTranslatorFactory())
                            .optEngine("PyTorch")
                            .optArgument("maxLength", "256")
                            .build();

[legalizenet]

@frankfliu - Thanks as expected it was the max size. yes adding maxLength = 256 works now. Problem solved for all-roberta model .. BUT..

So along the same line if I take the same long text from above and run it against bert-base-uncased model using DJL I get following error

RuntimeError: The expanded size of the tensor (542) must match the existing size (512) at non-singleton dimension 1. Target sizes: [1, 542]. Tensor sizes: [1, 512]

I have tried setting maxLength = 256 or 512, still does not work..

How to fix that?.

[frankfliu]

@legalizenet

Can show me the link the model in huggingface? which bert-base-uncased?

Are you talking about djl://ai.djl.huggingface.pytorch/bert-base-cased? This is not a text embedding model, this is a fill mask model

[legalizenet]

@frankfliu , this one I downloaded end of last year .. I believe it was thru sentence transformers. When I looked at the model . this is what I see in the config.json

.cache/torch/sentence_transformers/bert-base-uncased

[frankfliu]

I cannot find it from huggingface hub. The error you got most likely is because the translator doesn't match the model.

Can you debug your python code and see what's the expected input shape?

[frankfliu]

@legalizenet

I think your tokenizer default TruncationPolicy is DONOT_TRUNCATE, you can manually set truncation:

.optArgument("truncation", "true")

I created a PR to force truncation by default: #3068

[legalizenet]

@frankfliu - that seems to work now. But that raises another question.

That means I have to force truncation only for bert model since Im seeing this error , for sbert (minilm, mpnet, roerta large) I do not see this error even if I pass longer text (with or without "maxLength" option)

Also your point about bert is not a text embedding model and its a fill mask model, is there any concern of using it with text embedding translator factory ?

[frankfliu]

@legalizenet

Let me try to clarify:

1. Tokenizer has a default truncation policy, some tokenizer set to DO_NOT_TRUNCATE as default. DJL was using tokenizer's default. Now we forcibly set to truncate as default, and you can override it if you want to.
2. With our latest code, if you use model from our model zoo, you don't need to set "maxLength" any more, we can get the maxLength from huggingface's configuration file.
3. I checked bert-base-uncased model, this is a text embedding model, but this is not come from huggingface hub, it's from torch.hub. We don't have this model in our model zoo. You have to trace the model and manually set "maxLength", it should work with 0.28.0-SNAPSHOT version with my PR

[legalizenet]

Thanks for the clarification , now that makes a whole lot of sense.