Skip to content

Latest commit

 

History

History
236 lines (191 loc) · 8.33 KB

deeprec.adoc

File metadata and controls

236 lines (191 loc) · 8.33 KB
Raw

DeepRec Project Proposal

Name of project:

DeepRec

Requested project maturity level:

Incubation

Project description

DeepRec is a high-performance recommendation deep learning framework based on TensorFlow.

Recommendation models have huge commercial values for areas such as retailing, media, advertisements, social networks and search engines. Unlike other kinds of models, recommendation models have large amount of non-numeric features such as id, tag, text and so on which lead to huge parameters.

DeepRec has been developed since 2016, which supports core businesses such as Taobao Search, recommendation and advertising. It precipitates a list of features on basic frameworks and has excellent performance in recommendation models training and inference. So far, in addition to Alibaba Group, dozens of companies have used DeepRec in their business scenarios.

DeepRec has super large-scale distributed training capability, supporting recommendation model training of trillion samples and over ten trillion parameters. For recommendation models, in-depth performance optimization has been conducted across CPU and GPU platform. It contains list of features to improve usability and performance for super-scale scenarios.

DeepRec is supported by Alibaba, AntGroup, Intel and NVIDIA.

Statement on alignment with LF AI’s mission

DeepRec provides AI developers, researchers and data scientists in recommendation, advertising and search with tooling to provide better AUC and higher performance.

Have you identified possible collaboration opportunities with current LF AI hosted projects

  • Horovod

  • Feast

  • ONNX

License name

Apache License 2.0

Source control (GitHub, etc.)

Issue tracker (GitHub, JIRA, etc)

Collaboration tools

  • DingTalk and WeChat (in use)

  • Hope to have maillist and Slack channel under LFAI.

External dependencies

  • absl-py (Apache License 2.0)

  • abseil-cpp (Apache License 2.0)

  • alibabacloud-oss-sdk (Apache License 2.0)

  • Apache Kafka (Apache License 2.0)

  • apple_swift_swift_protobuf (Apache License 2.0)

  • ARM_NEON_2_x86_SSE (Custom)

  • arrow (Apache License 2.0)

  • astor (BSD-3-Clause License)

  • astunparse (PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2)

  • backprots.weakref (PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2)

  • bazelbuild_rules_android (Apache License 2.0)

  • bazelbuild_rules_apple (Apache License 2.0)

  • bazelbuild_rules_swift (Apache License 2.0)

  • bazelbuild_apple_support (Apache License 2.0)

  • bazel-skylib (Apache License 2.0)

  • boost (Boost Software License)

  • brotli (MIT License)

  • bzip2 (Modified zlib License)

  • checkerframework_qual (Apache License 2.0)

  • colm (MIT License)

  • cuCollections (Apache License 2.0)

  • cudnn-frountend (MIT License)

  • ComputeLibrary (MIT License)

  • c-ares (MIT License)

  • cython (Apache License 2.0)

  • concurrentqueue (BSD License/Boost Software License/zlib License)

  • cub (BSD-3-Clause License)

  • curl (MIT License)

  • double-conversion (BSD-3-Clause License)

  • Eigen3 (Mozilla Public License 2.0)

  • enum34 (PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2)

  • farmhash (MIT License)

  • FlatBuffer (Apache License 2.0)

  • fft2d (Custom)

  • functools32 (PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2)

  • gast (MIT License)

  • gemmlowp (Apache License 2.0)

  • gflags (BSD-3-Clause License)

  • giflib (PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2)

  • googleapis (Apache License 2.0)

  • google-cloud-cpp (Apache License 2.0)

  • googletest (BSD-3-Clause License)

  • google_testing_compile (New BSD License)

  • google_truth (Apache License 2.0)

  • google_pprof (Apache License 2.0)

  • grpc (Apache License 2.0)

  • Horovod (Apache License 2.0)

  • hamcrest-core (New BSD License)

  • hiredis (BSD-3-Clause license)

  • hwloc (BSD License)

  • hugectr (Apache License 2.0)

  • Intel-TensorFlow (Apache License 2.0)

  • jsoncpp (Public Domain/MIT License)

  • junit (Common Public License Version 1.0)

  • leveldb (BSD-3-Clause License)

  • libaio (GPLv2 License)

  • libarp1 (Apache License 2.0)

  • libarputil1 (Apache License 2.0)

  • libcuckoo (Apache License 2.0)

  • libevent (BSD License)

  • libexpat (MIT License)

  • libpng (PNG Reference Library License)

  • librdkafka (MIT License)

  • libxsmm (BSD-3-Clause License)

  • libz4 (BSD 2-Clause License/GPLv2 License)

  • linenoise (BSD-2-Clause License)

  • llvm (Apache License 2.0)

  • lmdb (OpenLDAP Public License)

  • lz4 (BSD 2-Clause license/GPLv2 license)

  • MKL-DNN (Apache License 2.0)

  • MLIR (Apache License 2.0)

  • mxml (Apache License 2.0)

  • NCCL (BSD License)

  • nanopb (Zlib License)

  • nelhage_rules_boost (Apache License 2.0)

  • ngraph (Apache License 2.0)

  • ngraph-tf (Apache License 2.0)

  • nlohmann_json (MIT License)

  • nsync (Apache License 2.0)

  • numpy (BSD-3-Clause License)

  • Nvidia-TensorFlow (Apache License 2.0)

  • NVTX (Apache License 2.0)

  • oneDNN (Apache License 2.0)

  • oneTBB (Apache License 2.0)

  • openssl (OpenSSL License/SSLeay License, BSD style License)

  • opt_einsum (MIT License)

  • pcre (MIT License)

  • ProtoBuf (BSD License)

  • pybind11 (BSD-style License)

  • rapidjson (MIT License)

  • ragel (MIT License)

  • re2 (BSD-3-Clause License)

  • rpi-newer-crosstools (None)

  • rocPRIM (MIT License)

  • seastar (Apache License 2.0)

  • sctp (GNU General Public License v2.0)

  • six (MIT License)

  • sparsehash-c11 (BSD-3-Clause License)

  • sqllite (Public Domain)

  • squareup_javapoet (Apache License 2.0)

  • snappy (BSD 2-Clause License)

  • swig (GPLv3 License)

  • TensorFlow (Apache License 2.0)

  • termcolor (PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2)

  • thrift (Apache License 2.0)

  • UUID (LGPL header only)

  • wrapt (BSD-2-Clause License)

  • xctestrunner (Apache License 2.0)

  • xfs (LGPL header only)

  • xsimd (BSD-3-Clause License)

  • zlib (zlib License)

  • zstd (BSD License)

Initial committers

Have the project defined the roles of contributor, committer, maintainer, etc.

This is a part of the governance document.

Total number of contributors to the project including their affiliations.

Does the project have a release methodology

Artifacts of a release

Did the project achieve any of the CII best practices badges?

None yet

Do you have any specific infrastructure requests needed as part of hosting the project in the LF AI

Currently DeepRec’s CICD is running on Alibaba Cloud.

Project website

None yet

Social media accounts

  • Twitter: None yet

  • Youtube: None yet

Existing sponsorship

Alibaba, AntGroup, Intel, NVIDIA have provided developer resources to improve DeepRec.