Create new translator for other Bert models

[FatosTorba]

Question

How to create translator for other Bert models ?
Thank you :)

[zachgk]

The translator describes how to pre-process and post-process your model. To create a translator, you need to figure out what format is expected by your BERT model and what format the output it produces is in. Then, you just need to convert into that format from whatever Java type your naturally have and what Java type you would like to use for the results.

Is there a particular BERT model you want to create a translator for?

[FatosTorba]

I would like to deploy BERT model to predict masked word in a sentence

[stu1130]

@FatosTorba we are working on common BERT translators and the tokenizer which uses https://github.com/google/sentencepiece. We will let you know when it is done. Meanwhile, you can check how to inference your own bert model for question answering for reference.

[allahbaksh]

This is good. I am going to try it with TensorFlow. I was getting an error with the Tensflow because of Input/Output share for SQUD. I don't think we have appropriate packages for the same? Do you have any thing which is inline with input/output shape of this https://miro.medium.com/max/1192/0*1N9VOiOYeunX2Y1Z
If I just use the present classes then with TensforFlow backend and pretrained SQUD bert model with the format of question.json defined as per the image. The system throws up error "Expects arg[0] to be string but int64 is provided"
Do we need to write our own Translator and data conversion for this? Do you have any suggestions?

[uniquetrij]

I'm trying to run BertQA using model trained on Tensorflow 2 and exported as saved model. The model loads properly and shows correct input/output tensors with model.describeInput() and model.describeOutput() methods. But I'm getting the following error while running model.newPredictor().predict(...).

org.tensorflow.exceptions.TFInvalidArgumentException: unique_ids_1:0 is both fed and fetched.
    at org.tensorflow.internal.c_api.AbstractTF_Status.throwExceptionIfNotOK (AbstractTF_Status.java:87)
    at org.tensorflow.Session.run (Session.java:595)
    at org.tensorflow.Session.access$100 (Session.java:70)
    at org.tensorflow.Session$Runner.runHelper (Session.java:335)
    at org.tensorflow.Session$Runner.run (Session.java:285)
    at ai.djl.tensorflow.engine.TfSymbolBlock.forward (TfSymbolBlock.java:121)
    at ai.djl.nn.Block.forward (Block.java:117)
    at ai.djl.inference.Predictor.predict (Predictor.java:117)
    at ai.djl.inference.Predictor.batchPredict (Predictor.java:157)
    at ai.djl.inference.Predictor.predict (Predictor.java:112)
    at com.mycompany.app.App.main (App.java:117)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:834)

The same model runs fine on python. Any suggestions?

[roywei]

Hi @uniquetrij what translators are you using for your Bert model? You may need to modify existing BertQA Translator to preprocess the input to the exact format that the model requires. Could you share your reproducible code, model, and output of describeInput and describeOutput ? Thanks

Here is an example of Bert Translator for TensorFlow 2 model, but it's for classification, it takes a single string input, you may need to modify it to take paragraph and questions (QAInput). It's an implementation of tokenization.py from google-research/bert.

[uniquetrij]

@roywei The issue seemed to be with the unique_ids tensor node for the model generated in run_squad.py from google-research/bert. The same tensor is being fed and fetched for input and result respectively in the model, hence the error. Strangely, the exported saved-model works perfectly when inferencing with python. I suspect the issue should be with the java wrapper for tensorflow being used with djl.

I was able to get around the error by changing line 672 in run_squad.py :

...
predictions = {
          "unique_ids": unique_ids,
          "start_logits": start_logits,
          "end_logits": end_logits,
      }
...

to

...
predictions = {
          "label_ids": tf.identity(unique_ids),
          "start_logits": start_logits,
          "end_logits": end_logits,
      }
...

and line 1260 (to prevent the code from breaking in run_squad.py):

...
unique_id = int(result["unique_ids"])
...

to

...
unique_id = int(result["label_ids"])
...

Exporting this to saved-model and then using it with DJL ran perfectly without any errors. I'll try to share an example soon. The modified version of the file can be found here. Note that this was forked from ultrons' repo to run on TF-v2. Also, I've added options for exporting a trained checkpoint to saved model in this file, thanks to bigboNed3.

[clausagerskov]

wow this change allowed me to successfully get predictions from a bert model deployed on vertex ai

[roywei]

Thanks! looks like a bug from TF Java side when invoking the C API. We will report and look into it.