Skip to content

This is the official repo for "Differentiable Model Scaling using Differentiable Topk"

Notifications You must be signed in to change notification settings

LKJacky/Differentiable-Model-Scaling

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Differentiable Model Scaling using Differentiable Topk

This repo is the official implementation of "Differentiable Model Scaling using Differentiable Topk".

Introduction

In this project, we propose a differentiable topk operator. It searches for k in a fully smooth and differentiable way. We use this operator to scale the model width and depth. We show that our method is more efficient and effective than the existing methods.

Getting Started

We provide the code for our differentiable topk operator and the experiments in the paper. We place the core code of the differentiable topk operator in the dms folder. The experiments are in the applications folder. Besides, our code is based on mmrazor, a powerful model compression toolbox. We provide a modified version of mmrazor in the mmrazor folder.

There are several steps to prepare the environment and run the codes.

  1. run pip install -e . to install differentiable topk operator and related modules.
  2. run cd mmrazor && pip install -e . to install the modified version of mmrazor.
  3. cd to the applications folder and find the experiment you want to run. Moreover, you also should follow the instructions in each experiment folder.

About

This is the official repo for "Differentiable Model Scaling using Differentiable Topk"

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published