Keras -> SNN IPython notebook

[studywolf]

Have an initial draft up, looking for feedback. Right now it's a fair bit longer than other examples.

• Where should I upload the pre-trained weights for the user to download and skip training themselves? (currently just ??? in the notebook)
• I'm not pumped on having test_images_30 and test_images_100, but I didn't come up with a much better way to do generate and pass around two data that is presented for two different lengths of time. It'd be nice to make sure the user can run things out of order after the first few cells.
• I separated the plotting into 3 functions because it felt a bit cleaner, but could put them all back in the run_network function because they're not used anywhere else.

[hunse]

I would put the pre-trained weights on Google Drive.

[studywolf]

looks really good! a couple of smaller comments, in the conversion to spikes block there's non-zero outpt from the second layer. My understanding is this in from a bias term added to the final output, i think it'd be worth mentioning

In the naive conversion above we are getting random accuracy (~10%). And if we look at the neural activity, we can see why: the activities are all zero! (The non-zero output in the second layer we are seeing is a result of the bias term added to the output node.) Referring ...

In the synaptic smoothing block: mentioning that the synapse is averaging over a short time (not necessary for those familiar with low pass filters but just a bit more explicit for others)

Intuitively, we can think of this as computing a running average of each neuron's activity over a short window of time (rather than just looking at the spikes on the last timestep)

In the firing rates block, the line about the true firing rate of the neuron I found confusing, attempted a rewrite

Another way that we can improve network performance is by increasing the firing rates of the neurons. Neurons that spike more frequently update their output signal more often. This means that as firing rates increase the behaviour of the spiking model will more closely match the original non-spiking model (where the neuron is directly outputting its true firing rate every timestep).

And in the regularization during training block, I think it's worth mentioning that adding the term during training works is also what you want to do when using nonlinear activation functions

... as applying L2 regularization to the firing rates, but we've shifted the regularization point from 0 to some target value. One of the benefits of this method is that it is also effective for neurons with non-linear activation functions, such as the LIF neuron.

[drasmuss]

Yeah those are all good clarifications, made the changes

[hunse]

Looks good. I made a few changes. The first commit makes a few minor wording changes, and also decreases the batch size during testing to 20. I first started looking at batch size here because the synapse runs (with n_steps = 150) were giving me OOM errors, and my GPU is not small (6 GB), so I wanted to decrease it to make sure we work across a variety of GPUs. Then I realized that it actually runs considerably faster for me with a smaller batch size. 20 seemed like a good balance.

The second commit adds a line that shows the "output density" (number of non-zero timesteps) to run_network. The idea behind this is it's nice to have some sort of quantitative measure of the firing rate, particularly when we get into changing the rate scaling. However, we can't just take the mean output, because rate scaling uses the neuron amplitude to counteract the increase in firing rate, so the output stays the same. So output density seemed like a better metric. I think it makes the discussion around firing rates a bit clearer (particularly when we say things like "increasing the scaling too high makes the network like a rate network again"), but I'm open to suggestions.

[drasmuss]

Looks good! I thought that the notion of "output density" might be a bit too abstract for people, so I changed it to just directly report firing rates. I also added it directly to the title of the neural activities plot, so that the connection between that graph and the reported numbers was clear.

[hunse]

One more thought: Maybe in the section that does firing rate scaling, we should mention that this only works for ReLUs, because they're scale invariant, and not for LIFs. This would also be a nice segue into the regularization section, where we can basically say "here's a method that can work with any neuron type, including LIFs".

[studywolf]

that was the original flow, i think that it was removed because we then didn't show any training with LIFs?

[hunse]

Is there a reason not to have the regularization section use LIFs?

[drasmuss]

I looked at it briefly, but it changes some other things as well (e.g. the neural activities look a bit different, the shape of the output predictions gets sharper thanks to the LIF nonlinearity). So you have to discuss that a bit (to make it clear that this is a result of the different neuron type, not the regularization). And the accuracy isn't as good, so then you have to tweak some things for that. So I thought that all ended up just being distracting from the main point we wanted to make (that you can use a regularization function to control the firing rates).

[studywolf]

so i find it super useful to limit memory allocation

# Set up the TF memory allocation settings
config = tf.compat.v1.ConfigProto()
# Don't pre-allocate memory, allocate as needed
config.gpu_options.allow_growth = True
# Only allow a total of half the GPU memory to be allocated
config.gpu_options.per_process_gpu_memory_fraction = 0.5
# Create a session with the above options specified
tf.compat.v1.keras.backend.set_session(tf.compat.v1.Session(config=config))

otherwise i find i have to shut down the notebook before i can run any other models. what are the thoughts on this?

[drasmuss]

You can set that when you start up the notebook like

TF_FORCE_GPU_ALLOW_GROWTH=true jupyter notebook

In general, I think those kind of environmental configuration things should be left up to the user, rather than set in our examples. I suspect 90% of users will just be looking at the rendered examples online, not running it themselves, so it'd just add clutter for them.