v0.4.0 (#137)

* Change README logo if in dark mode (#95) * Start transform tests & minor `RandomTimeShift` optimization (#94) * Start transform tests & slight optimization in RandomTimeShift * Add new length check for TimeCrop tests * Code style * Fixed name mangling. * QAM/PSK Pulse shaping filter transition bandwidth corrected (#98) * excess bandwidth is defined in porportion to signal bandwidth, not sampling rate, thus needs to be scaled by the samples per symbol * filling in a comment to describe modification to code * QAM/PSK Pulse shaping filter transition bandwidth corrected (#98) * excess bandwidth is defined in porportion to signal bandwidth, not sampling rate, thus needs to be scaled by the samples per symbol * filling in a comment to describe modification to code * OFDM Modulator filter lengths estimated and bandwidth randomized (#99) * * cutoff frequency for LPF now randomized when using 'rand_lpf' * derives a transition bandwidth from the cutoff frequency * uses filter length approximating function for the randomized LPF * using filter estimation function for pre-computed LPF taps * Tests for visual inspection. (#103) * 91 create generation performance benchmarks for each modulation type (#104) * Initial benchmarking code. * Some benchmarks * Adding initial benchmarks. * Fix action. --------- * 75 examine ofdm generation for potential speedups for sig53 (#105) * Tests for visual inspection of modulation generation. (#102) * Optimizations show significant improvement in generation speed. * Nominal behavior after using scipy. * Adding initial Dockerfile (#108) * Incrementing version * Fix float issue (#111) * Initial draft of restructuring transforms (#106) * Flatten transforms to mirror torchvision/audio structure & add reprs * Address flake8 errors * Update transform imports with restructuring * Formatting. --------- * Fixing broken examples, formatting files, etc. * Adjustments for generation speed * Wideband generation working. * 45 consider compatibility with torch 20 (#115) * Seems to work * Workflow is broken * Python 3.7 not supported for Torch 2.0 * Adjusting test workflow * Extracted out a method for filter design. Put DSP-only things in utils/dsp (#116) * Benchmark and visualize wideband dataset generation. (#118) * Extracted out a method for filter design. Put DSP-only things in utils/dsp * New tests. * Migrated to pytest. (#119) * Added model instantiation tests for narrowband signals. (#120) * 85 gmskgfsk also needs faster filtering due to convolution of long signals with gaussian pulse shape (#121) * Change all references to convolution to scipy-based convolutions. * Missed one. * Add mypy workflow check and fix all mypy-found bugs (#123) * Fix mypy in target_transforms * Add mypy workflow for static type checking * Fix typo in mypy.yml workflow * Fix mypy in torchsig/transforms/functional.py * Fix mypy in utils/types.py * Fix mypy in torchsig/utils/ * Fix mypy in torchsig/transforms/ * Fix mypy in torchsig/datasets/ * Fix mypy in torchsig/models/ * Format with pyfmt * Fix isinstance(x, Callable) * Add PR template * Adjusting module (#130) * 131 configure package for pypi release (#132) * Adjusting installation * More information on pyproject.toml * More information on pyproject.toml * I guess some keywords in current documentation do not work :/. * Ignore distributable artifacts * More documentation and examples (#134) * More documentation and examples * Added script to train * Remove duplicate script * 126 create dockerfile and script for generating all versions of widebandsig53 (#135) * Added generation script. * Adding scripts * Fixing mypy issue. * 127 create test suite for visually validating transforms (#136) * Some transforms here * Most transforms included --------- Co-authored-by: lboegner <[email protected]> Co-authored-by: Garrett Vanhoy <[email protected]> Co-authored-by: MattCarrickPL <[email protected]>
TorchDSP · Jun 13, 2023 · e9fa58f · e9fa58f
1 parent 38cd9fd
commit e9fa58f
Show file tree

Hide file tree

Showing 77 changed files with 9,034 additions and 8,521 deletions.
diff --git a/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md
@@ -0,0 +1,24 @@
+## Summary
+
+Describe your changes here specifying if the change is a bug fix, enhancement, new feature, etc.
+
+## Test Plan
+
+Describe how you tested and verified your changes here (changes captured in existing tests, built and ran new tests, etc.).
+
+## Before Submitting
+- [ ] Check mypy locally
+    - `pip3 install mypy==1.2.0`
+    - `mypy --ignore-missing-imports torchsig`
+    - Address any error messages
+- [ ] Lint check locally
+    - `pip3 install flake8`
+    - `flake8 --select=E9,F63,F7,F82 torchsig`
+    - Address any error messages
+- [ ] Run formatter if needed
+    - `pip3 install git+https://github.com/GooeeIOT/pyfmt.git`
+    - `pyfmt torchsig`
+- [ ] Run test suite locally
+    - `pytest --ignore-glob=*_figures.py --ignore-glob=*_benchmark.py`
+    - Ensure tests are successful prior to submitting PR
+
diff --git a/.github/workflows/mypy.yml b/.github/workflows/mypy.yml
@@ -0,0 +1,30 @@
+name: mypy
+
+on: [push]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.8", "3.10"]
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Install Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - uses: actions/cache@v3
+        with:
+          path: ${{ env.pythonLocation }}
+          key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
+      - name: Build package
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install .
+      - name: Static type check with mypy
+        run: |
+          pip install mypy==1.2.0
+          mypy --ignore-missing-imports torchsig
+
diff --git a/.github/workflows/pip_build.yml b/.github/workflows/pip_build.yml
@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.7", "3.10"]
+        python-version: ["3.8", "3.10"]
 
     steps:
       - uses: actions/checkout@v3
@@ -21,4 +21,5 @@ jobs:
           key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
       - name: Build Package
         run: |
+          python -m pip install --upgrade pip
           python -m pip install .
diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml
@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.7", "3.10"]
+        python-version: ["3.8", "3.10"]
 
     steps:
       - uses: actions/checkout@v3
@@ -21,6 +21,7 @@ jobs:
           key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
       - name: Build package
         run: |
+          python -m pip install --upgrade pip
           python -m pip install .
       - name: Lint with flake8
         run: |

diff --git a/.gitignore b/.gitignore
@@ -11,3 +11,4 @@ lightning_logs/
 *.pt
 *.jpg
 *.benchmarks/
+dist/
diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime
+FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime
 
 ENV DEBIAN_FRONTEND=noninteractive
 

diff --git a/README.md b/README.md
@@ -43,6 +43,35 @@ cd torchsig
 pip install .
 ```
 
+## Generating the Datasets
+If you'd like to generate the named datasets without messing with your current Python environment, you can build the development container and use it to generate data at the location of your choosing.
+
+```
+docker build -t torchsig -f Dockerfile .
+docker run -u $(id -u ${USER}):$(id -g ${USER}) -v `pwd`:/workspace/code/torchsig torchsig python3 torchsig/scripts/generate_sig53.py --root=/workspace/code/torchsig/data --all=True
+```
+
+For the wideband dataset, you can do:
+
+```
+docker build -t torchsig -f Dockerfile .
+docker run -u $(id -u ${USER}):$(id -g ${USER}) -v `pwd`:/workspace/code/torchsig torchsig python3 torchsig/scripts/generate_wideband_sig53.py --root=/workspace/code/torchsig/data --all=True
+```
+
+If you do not need to use Docker, you can also just generate using the regular command-line interface
+
+```
+python3 torchsig/scripts/generate_sig53.py --root=torchsig/data --all=True
+```
+
+or for the wideband dataset:
+
+```
+python3 torchsig/scripts/generate_wideband_sig53.py --root=torchsig/data --all=True
+```
+
+Then, be sure to point scripts looking for ```root``` to ```torchsig/data```.
+
 ## Using the Dockerfile
 If you have Docker installed along with compatible GPUs and drivers, you can try:
 

diff --git a/examples/00_example_sig53_dataset.py b/examples/00_example_sig53_dataset.py
@@ -93,6 +93,7 @@ def __len__(self) -> int:
 
 data_loader = DataLoader(dataset=plot_dataset, batch_size=16, shuffle=True)
 
+
 # Transform the plotting titles from the class index to the name
 def target_idx_to_name(tensor: np.ndarray) -> list:
     batch_size = tensor.shape[0]

diff --git a/examples/01_example_modulations_dataset.py b/examples/01_example_modulations_dataset.py
@@ -160,7 +160,7 @@ def __len__(self) -> int:
 # ### Save Data to LMDB
 # As a final exercise for this example notebook, the dataset can be saved to an LMDB static dataset for offline use. Note this is similar to how the static Sig53 dataset is generated and saved to serve as a static performance evaluation dataset.
 
-env = lmdb.open("dataset", max_dbs=3 if include_snr else 2, map_size=int(1e12))
+env = lmdb.open("examples/dataset", max_dbs=3 if include_snr else 2, map_size=int(1e12))
 
 iq_sample_db = env.open_db("iq_samples".encode())
 modulations_db = env.open_db("modulations".encode())

diff --git a/examples/02_example_sig53_classifier.py b/examples/02_example_sig53_classifier.py
@@ -5,17 +5,19 @@
 # ### Import Libraries
 # First, import all the necessary public libraries as well as a few classes from the `torchsig` toolkit. An additional import from the `cm_plotter.py` helper script is also done here to retrieve a function to streamline plotting of confusion matrices.
 
-from torchsig.transforms.target_transforms.target_transforms import DescToClassIndex
+from torchsig.transforms.target_transforms import DescToClassIndex
 from torchsig.models.iq_models.efficientnet.efficientnet import efficientnet_b4
 from torchsig.utils.writer import DatasetCreator
-from torchsig.transforms.wireless_channel.wce import RandomPhaseShift
-from torchsig.transforms.signal_processing.sp import Normalize
-from torchsig.transforms.expert_feature.eft import ComplexTo2D
-from torchsig.transforms.transforms import Compose
+from torchsig.transforms.transforms import (
+    RandomPhaseShift,
+    Normalize,
+    ComplexTo2D,
+    Compose,
+)
 from pytorch_lightning.callbacks import ModelCheckpoint
 from pytorch_lightning import LightningModule, Trainer
 from sklearn.metrics import classification_report
-from cm_plotter import plot_confusion_matrix
+from torchsig.utils.cm_plotter import plot_confusion_matrix
 from torchsig.datasets.sig53 import Sig53
 from torchsig.datasets.modulations import ModulationsDataset
 from torch.utils.data import DataLoader
@@ -112,17 +114,17 @@
 # ### Format Dataset for Training
 # Next, the datasets are then wrapped as `DataLoaders` to prepare for training.
 
-# Create dataloaders
+# Create dataloaders"data
 train_dataloader = DataLoader(
     dataset=sig53_clean_train,
-    batch_size=16,
+    batch_size=8,
     num_workers=8,
     shuffle=True,
     drop_last=True,
 )
 val_dataloader = DataLoader(
     dataset=sig53_clean_val,
-    batch_size=16,
+    batch_size=8,
     num_workers=8,
     shuffle=False,
     drop_last=True,

diff --git a/examples/03_example_widebandsig53_dataset.ipynb b/examples/03_example_widebandsig53_dataset.ipynb
diff --git a/examples/04_example_wideband_modulations_dataset.ipynb b/examples/04_example_wideband_modulations_dataset.ipynb
@@ -1,6 +1,7 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -11,6 +12,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -24,17 +26,16 @@
    "outputs": [],
    "source": [
     "from torchsig.utils.visualize import MaskClassVisualizer, mask_class_to_outline, complex_spectrogram_to_magnitude\n",
-    "from torchsig.transforms.target_transforms.target_transforms import DescToMaskClass\n",
+    "from torchsig.transforms.target_transforms import DescToMaskClass\n",
     "from torchsig.datasets.wideband import WidebandModulationsDataset\n",
-    "from torchsig.transforms.expert_feature.eft import Spectrogram\n",
-    "from torchsig.transforms.signal_processing.sp import Normalize\n",
-    "from torchsig.transforms.transforms import Compose\n",
+    "from torchsig.transforms.transforms import Spectrogram, Normalize, Compose\n",
     "from torch.utils.data import DataLoader\n",
     "import matplotlib.pyplot as plt\n",
     "import numpy as np"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -74,6 +75,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {
     "tags": []
@@ -114,6 +116,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {
     "tags": []
@@ -186,7 +189,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.0"
+   "version": "3.8.10"
   }
  },
  "nbformat": 4,