For Mobilenet V4 large #821

freedomtan · 2023-11-28T09:03:01Z

Use Mobilenet V4 large for Image Classification V2.

github-actions · 2023-11-28T09:03:16Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

sonarcloud · 2023-11-28T10:37:06Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
5 Code Smells

0.0% Coverage
0.0% Duplication

freedomtan · 2024-01-02T13:35:50Z

@mohitmundhragithub I fixed a stupid bug for floating model. Please try again.

mohitmundhragithub · 2024-01-08T11:02:19Z

@mohitmundhragithub I fixed a stupid bug for floating model. Please try again.

Thanks a lot @freedomtan to share this.

I have added one more commit to modify the input images required for Calibration set for PTQ. Please have look.
@Mostelk, IMHO this removes the need for the default normalization model, which you have asked from Colby over email.

freedomtan · 2024-02-01T05:43:53Z

@anhappdev For MobileNet EdgeTPU, the input tensor is 224x224x3; for newly introduced model, it's 384x384x3. What's the best way to have both them? Add new dataset type, e.g., imagenet_384? Or, we should add width and height paramters?

anhappdev · 2024-02-01T06:54:37Z

@anhappdev For MobileNet EdgeTPU, the input tensor is 224x224x3; for newly introduced model, it's 384x384x3. What's the best way to have both them? Add new dataset type, e.g., imagenet_384? Or, we should add width and height paramters?

Where should we have both of them? Do you mean the image_classification_v2 task should have multiple models with different input sizes?

There is already an image_width and image_height parameter in ModelConfig.

mobile_app_open/flutter/cpp/proto/mlperf_task.proto

Lines 95 to 108 in e02bdc8

    
           message ModelConfig { 
        
             // unique ID used by UI to find matching description 
        
             required string id = 1; 
        
             // Human-readable name of the model. 
        
             required string name = 2; 
        
             // Offset value of the model if applicable. 
        
             optional int32 offset = 3 [default = 0]; 
        
             // image width of the input (if model takes images) 
        
             optional int32 image_width = 4; 
        
             // image height of the input (if model takes images) 
        
             optional int32 image_height = 5; 
        
             // Number of detection classes if applicable 
        
             optional int32 num_classes = 6; 
        
           }

I

freedomtan · 2024-02-01T08:46:03Z

@anhappdev For MobileNet EdgeTPU, the input tensor is 224x224x3; for newly introduced model, it's 384x384x3. What's the best way to have both them? Add new dataset type, e.g., imagenet_384? Or, we should add width and height paramters?

Where should we have both of them? Do you mean the image_classification_v2 task should have multiple models with different input sizes?

nope, image_classification and image_classification_v2 have different sizes.

There is already an image_width and image_height parameter in ModelConfig.

mobile_app_open/flutter/cpp/proto/mlperf_task.proto

Lines 95 to 108 in e02bdc8

message ModelConfig {

// unique ID used by UI to find matching description

required string id = 1;

// Human-readable name of the model.

required string name = 2;

// Offset value of the model if applicable.

optional int32 offset = 3 [default = 0];

// image width of the input (if model takes images)

optional int32 image_width = 4;

// image height of the input (if model takes images)

optional int32 image_height = 5;

// Number of detection classes if applicable

optional int32 num_classes = 6;

}

I

We hard-coded image size for imagenet dataset at https://github.com/mlcommons/mobile_app_open/blob/master/flutter/cpp/flutter/dart_run_benchmark.cc#L73-L77

  switch (in->dataset_type) {
    case ::mlperf::mobile::DatasetConfig::IMAGENET:
      dataset = std::make_unique<::mlperf::mobile::Imagenet>(
          backend.get(), in->dataset_data_path, in->dataset_groundtruth_path,
          in->dataset_offset, 224, 224 /* width, height */);
      break;
...

anhappdev · 2024-02-01T10:17:50Z

We hard-coded image size for imagenet dataset at ...

In that case, we just need to update the code to read the image size from the task file?

freedomtan · 2024-02-02T04:43:19Z

We hard-coded image size for imagenet dataset at ...

In that case, we just need to update the code to read the image size from the task file?

Yes, that's a better solution. A dirty hack is to add something like case ::mlperf::mobile::DatasetConfig::IMAGENET_384:

  switch (in->dataset_type) {
    case ::mlperf::mobile::DatasetConfig::IMAGENET:
      dataset = std::make_unique<::mlperf::mobile::Imagenet>(
          backend.get(), in->dataset_data_path, in->dataset_groundtruth_path,
          in->dataset_offset, 224, 224 /* width, height */);
      break;

How to update our code to pass width and height to the dart_ffi_run_benchmark_out* dart_ffi_run_benchmark in
https://github.com/mlcommons/mobile_app_open/blob/master/flutter/cpp/flutter/dart_run_benchmark.cc

anhappdev · 2024-02-02T05:15:38Z

Yes, that's a better solution.
How to update our code to pass width and height to the dart_ffi_run_benchmark_out* dart_ffi_run_benchmark in
https://github.com/mlcommons/mobile_app_open/blob/master/flutter/cpp/flutter/dart_run_benchmark.cc

I can send a PR to pass image_width and image_height to dart_ffi_run_benchmark. It requires code changes in multiple places.

anhappdev · 2024-02-02T05:19:27Z

@freedomtan Can you rebase this PR on the 719/new-edge-tpu-model and also change this PR to merge into 719/new-edge-tpu-model instead of master? I just updated 719/new-edge-tpu-model with the latest commits from master.

freedomtan · 2024-02-02T06:16:33Z

@freedomtan Can you rebase this PR on the 719/new-edge-tpu-model and also change this PR to merge into 719/new-edge-tpu-model instead of master? I just updated 719/new-edge-tpu-model with the latest commits from master.

sure, I'll do it.

anhappdev · 2024-02-02T06:34:29Z

feat: pass model config from tasks.pbtxt to benchmark run #853

fixed a stupid bug

freedomtan · 2024-02-13T04:25:44Z

@RSMNYS The CoreML model for MobilenetV4 Large converted TensorFlow doesn't perform as good as as expected, this could be used a case for optimizing CoreML models.

freedomtan · 2024-02-13T04:27:16Z

@anhappdev Any comments of how to the 6 failed tests?

freedomtan · 2024-02-13T04:40:55Z

Pixel backend: Tested on @freedomtan's Pixel 7a
tflite fallback: tested on a Pixel 4 by @freedomtan
MTK Neuron
iOS CoreML

anhappdev · 2024-02-13T04:46:38Z

@anhappdev Any comments of how to the 6 failed tests?

They mostly failed on expected accuracy or throughput. I will run them several times and update the expected values.

anhappdev · 2024-02-13T14:46:27Z

Currently, the test-android-apk-tflite and test-android-apk-pixel failed because accuracy is always 0.0.
I don't have a Pixel device to debug this. @freedomtan Can you take a look and run the test locally (with make flutter/test cmd) with your devices to test?

* Update S3 provider in android-build-test.yml * Update env var name --------- Co-authored-by: Anh <[email protected]>

freedomtan · 2024-02-19T15:50:10Z

flutter/test

sometimes it failed, sometimes all tests passed

tested on Pixel 7 Pro running latest Android 14.

an "all tests passed" example was uploaded to Firebase,

Uploaded to user/B0rnrmYyh9XMldU5CZuzRELFZRY2/result/2024-02-20T06-21-02_9b560a3c-41b8-40f5-b7b7-6f8526ee61e0.json

freedomtan · 2024-02-20T06:29:25Z

Let's test the TFLite backend on more devices to see if we can figure out why we got accuracy equals 0 on Pixel 5.

freedomtan · 2024-02-20T08:16:45Z

On a Samsung Galaxy S22+ (Exynos 2200), with TFLite backend only, I got expected accuracy numbers.

anhappdev · 2024-02-20T08:47:31Z

Pixel backend:

image_classification_v2:
- Pixel 6, API Level 33 > accuracy OK
- Pixel 6, API Level 32 > accuracy OK
- Pixel 6, API Level 31 > accuracy OK
image_classification_offline_v2:
- Pixel 6, API Level 33 > accuracy Zero
- Pixel 6, API Level 32 > accuracy Zero
- Pixel 6, API Level 31 > crash

TFLite backend:

image_classification_v2:
- Pixel 6, API Level 33 > accuracy OK
- Pixel 6, API Level 32 > accuracy OK
- Pixel 6, API Level 31 > accuracy OK
- Pixel 5, API Level 30 > accuracy Zero
image_classification_offline_v2:
- Pixel 6, API Level 33 > accuracy OK
- Pixel 6, API Level 32 > accuracy OK
- Pixel 6, API Level 31 > accuracy OK
- Pixel 5, API Level 30 > accuracy Zero

freedomtan · 2024-02-21T02:51:32Z

Pixel backend:

image_classification_v2:

Pixel 6, API Level 33 > accuracy OK

Pixel 6, API Level 32 > accuracy OK

Pixel 6, API Level 31 > accuracy OK

image_classification_offline_v2:

Pixel 6, API Level 33 > accuracy Zero

Pixel 6, API Level 32 > accuracy Zero

Pixel 6, API Level 31 > crash

TFLite backend:

image_classification_v2:

Pixel 6, API Level 33 > accuracy OK

Pixel 6, API Level 32 > accuracy OK

Pixel 6, API Level 31 > accuracy OK

Pixel 5, API Level 30 > accuracy Zero

image_classification_offline_v2:

Pixel 6, API Level 33 > accuracy OK

Pixel 6, API Level 32 > accuracy OK

Pixel 6, API Level 31 > accuracy OK

Pixel 5, API Level 30 > accuracy Zero

That's interesting. I guess for Pixel backend + image_classification_offline_v2, we can get expected accuracy by reducing batch size.

For Pixel 5, either API level or other software stack issues?

anhappdev · 2024-02-21T04:50:21Z

For Pixel 5, either API level or other software stack issues?

The Pixel 5 on Firebase Test Lab has only API Level 30 available, so I cannot test it with other API levels.
I tested on Motorola G20, API Level 30, TFLite backend > the accuracy is OK.
So I guess the issue is something specific to Pixel 5.

sonarcloud · 2024-02-21T05:25:59Z

Quality Gate passed

Issues
0 New issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

RSMNYS · 2024-02-22T17:10:58Z

@RSMNYS The CoreML model for MobilenetV4 Large converted TensorFlow doesn't performance as good as as expected, this could be used a case for optimizing CoreML models.

@freedomtan is this model the one we need to work with: https://github.com/mlcommons/mobile_open/raw/main/vision/mobilenetV4?

or this savedModel: https://github.com/mlcommons/mobile_open/releases/tag/model_upload?

freedomtan · 2024-02-23T02:10:54Z

@RSMNYS The CoreML model for MobilenetV4 Large converted TensorFlow doesn't performance as good as as expected, this could be used a case for optimizing CoreML models.

@freedomtan is this model the one we need to work with: https://github.com/mlcommons/mobile_open/raw/main/vision/mobilenetV4?

or this savedModel: https://github.com/mlcommons/mobile_open/releases/tag/model_upload?

Either one, or say, neither of them. We convert from the saved_model to CoreML model. The performance of the converted CoreML model is not as good as expected. Please check how we can get a better CoreML model from either tflite of saved_model.

RSMNYS · 2024-03-04T13:42:10Z

@RSMNYS The CoreML model for MobilenetV4 Large converted TensorFlow doesn't performance as good as as expected, this could be used a case for optimizing CoreML models.

@freedomtan is this model the one we need to work with: https://github.com/mlcommons/mobile_open/raw/main/vision/mobilenetV4?
or this savedModel: https://github.com/mlcommons/mobile_open/releases/tag/model_upload?

Either one, or say, neither of them. We convert from the saved_model to CoreML model. The performance of the converted CoreML model is not as good as expected. Please check how we can get a better CoreML model from either tflite of saved_model.

@freedomtan are we expecting to have the MobilenetV4 performance on the task image_classification_V2 similar as for the https://github.com/mlcommons/mobile_models/raw/main/v3_0/CoreML/MobilenetEdgeTPU.mlmodel which Is used for the image_classification task (currently near 1000 qps)?

freedomtan · 2024-03-05T05:08:10Z

@RSMNYS The CoreML model for MobilenetV4 Large converted TensorFlow doesn't performance as good as as expected, this could be used a case for optimizing CoreML models.

@freedomtan is this model the one we need to work with: https://github.com/mlcommons/mobile_open/raw/main/vision/mobilenetV4?
or this savedModel: https://github.com/mlcommons/mobile_open/releases/tag/model_upload?

Either one, or say, neither of them. We convert from the saved_model to CoreML model. The performance of the converted CoreML model is not as good as expected. Please check how we can get a better CoreML model from either tflite of saved_model.

@freedomtan are we expecting to have the MobilenetV4 performance on the task image_classification_V2 similar as for the https://github.com/mlcommons/mobile_models/raw/main/v3_0/CoreML/MobilenetEdgeTPU.mlmodel which Is used for the image_classification task (currently near 1000 qps)?

as far as I can tell, 1000 qps is too good to be true for Mobilenet V4 Large. But 300 qps should be possible, I guess.

freedomtan mentioned this pull request Nov 28, 2023

Test potential new models #822

Closed

Mostelk marked this pull request as ready for review November 29, 2023 04:48

Mostelk requested review from anhappdev and a team as code owners November 29, 2023 04:48

freedomtan marked this pull request as draft December 4, 2023 03:08

freedomtan force-pushed the test_new_model_l branch from 49d73aa to 07c1213 Compare February 2, 2024 06:27

anhappdev changed the base branch from master to 719/new-edge-tpu-model February 2, 2024 06:33

freedomtan force-pushed the test_new_model_l branch from 3570a1a to fd82641 Compare February 2, 2024 07:05

freedomtan and others added 9 commits February 2, 2024 15:28

hacks for model small

08bc86b

small -> medium

8a9e510

medium -> large

cf3fa94

for quantized models

4bea665

make float model work again

2283422

fixed a stupid bug

fix pixel and neuron settings

7317b9b

change preprocessing and label offset

bb7dda6

Adding changes for calibration dataset preprocessing

c259333

hacks

c15f4b0

Use MobileNetV4 model in Apple devices

fe4d2d1

Fix tflite_settings_windows.pbtxt

d15f5a8

Update expected_accuracy

dc42a63

anhappdev added 4 commits February 13, 2024 14:17

Update expected_accuracy and expected_throughput for Apple devices

f980d58

Use fp32 model for Windows

441cbf7

Update expected_throughput for Windows

73dc658

Update expected_throughput for Android

11e6c4c

Update S3 provider in android-build-test.yml (#855)

5c2717d

* Update S3 provider in android-build-test.yml * Update env var name --------- Co-authored-by: Anh <[email protected]>

anhappdev added 2 commits February 21, 2024 11:52

Reduce batch_size for image_classification_offline_v2 in Pixel 6

ce5e74a

Merge remote-tracking branch 'origin/master' into test_new_model_l

8b39846

anhappdev marked this pull request as ready for review February 27, 2024 06:19

anhappdev merged commit c128476 into 719/new-edge-tpu-model Feb 27, 2024
19 of 21 checks passed

anhappdev deleted the test_new_model_l branch February 27, 2024 06:57

github-actions bot locked and limited conversation to collaborators Feb 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

For Mobilenet V4 large #821

For Mobilenet V4 large #821

freedomtan commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023 •

edited

Loading

sonarcloud bot commented Nov 28, 2023

freedomtan commented Jan 2, 2024

mohitmundhragithub commented Jan 8, 2024 •

edited

Loading

freedomtan commented Feb 1, 2024 •

edited

Loading

anhappdev commented Feb 1, 2024 •

edited

Loading

freedomtan commented Feb 1, 2024

anhappdev commented Feb 1, 2024

freedomtan commented Feb 2, 2024 •

edited

Loading

anhappdev commented Feb 2, 2024

anhappdev commented Feb 2, 2024

freedomtan commented Feb 2, 2024

anhappdev commented Feb 2, 2024

freedomtan commented Feb 13, 2024 •

edited

Loading

freedomtan commented Feb 13, 2024

freedomtan commented Feb 13, 2024 •

edited

Loading

anhappdev commented Feb 13, 2024

anhappdev commented Feb 13, 2024

freedomtan commented Feb 19, 2024 •

edited

Loading

freedomtan commented Feb 20, 2024

freedomtan commented Feb 20, 2024

anhappdev commented Feb 20, 2024 •

edited

Loading

freedomtan commented Feb 21, 2024

anhappdev commented Feb 21, 2024

sonarcloud bot commented Feb 21, 2024

RSMNYS commented Feb 22, 2024 •

edited

Loading

freedomtan commented Feb 23, 2024

RSMNYS commented Mar 4, 2024

freedomtan commented Mar 5, 2024

For Mobilenet V4 large #821

For Mobilenet V4 large #821

Conversation

freedomtan commented Nov 28, 2023 • edited Loading

github-actions bot commented Nov 28, 2023 • edited Loading

sonarcloud bot commented Nov 28, 2023

freedomtan commented Jan 2, 2024

mohitmundhragithub commented Jan 8, 2024 • edited Loading

freedomtan commented Feb 1, 2024 • edited Loading

anhappdev commented Feb 1, 2024 • edited Loading

freedomtan commented Feb 1, 2024

anhappdev commented Feb 1, 2024

freedomtan commented Feb 2, 2024 • edited Loading

anhappdev commented Feb 2, 2024

anhappdev commented Feb 2, 2024

freedomtan commented Feb 2, 2024

anhappdev commented Feb 2, 2024

freedomtan commented Feb 13, 2024 • edited Loading

freedomtan commented Feb 13, 2024

freedomtan commented Feb 13, 2024 • edited Loading

anhappdev commented Feb 13, 2024

anhappdev commented Feb 13, 2024

freedomtan commented Feb 19, 2024 • edited Loading

freedomtan commented Feb 20, 2024

freedomtan commented Feb 20, 2024

anhappdev commented Feb 20, 2024 • edited Loading

freedomtan commented Feb 21, 2024

anhappdev commented Feb 21, 2024

sonarcloud bot commented Feb 21, 2024

Quality Gate passed

RSMNYS commented Feb 22, 2024 • edited Loading

freedomtan commented Feb 23, 2024

RSMNYS commented Mar 4, 2024

freedomtan commented Mar 5, 2024

freedomtan commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023 •

edited

Loading

mohitmundhragithub commented Jan 8, 2024 •

edited

Loading

freedomtan commented Feb 1, 2024 •

edited

Loading

anhappdev commented Feb 1, 2024 •

edited

Loading

freedomtan commented Feb 2, 2024 •

edited

Loading

freedomtan commented Feb 13, 2024 •

edited

Loading

freedomtan commented Feb 13, 2024 •

edited

Loading

freedomtan commented Feb 19, 2024 •

edited

Loading

anhappdev commented Feb 20, 2024 •

edited

Loading

RSMNYS commented Feb 22, 2024 •

edited

Loading