Distributional Reinforcement Learning with Quantile Regression

[yydxlv]

Hi, what does the "u" means in the following code snippets? It seems that the "u" is not defined in the code? Thanks!

huber_loss = 0.5 * u.abs().clamp(min=0.0, max=k).pow(2)
huber_loss += k * (u.abs() - u.abs().clamp(min=0.0, max=k))
quantile_loss = (tau - (u < 0).float()).abs() * huber_loss

[hohoCode]

I think probably it should be something like:

u = dist - expected_quant

[angmc]

After adding u = dist - expected_quant

TypeError Traceback (most recent call last)
in ()
15
16 if len(replay_buffer) > batch_size:
---> 17 loss = compute_td_loss(batch_size)
18 losses.append(loss.data[0])
19

in compute_td_loss(batch_size)
17 huber_loss = 0.5 * u.abs().clamp(min=0.0, max=k).pow(2)
18 huber_loss += k * (u.abs() - u.abs().clamp(min=0.0, max=k))
---> 19 quantile_loss = (tau - (u < 0).float()).abs() * huber_loss
20 loss = quantile_loss.sum() / num_quant
21

/home/--/anaconda2/envs/tensorflow4/lib/python2.7/site-packages/torch/tensor.pyc in sub(self, other)
310
311 def sub(self, other):
--> 312 return self.sub(other)
313
314 def rsub(self, other):

TypeError: sub received an invalid combination of arguments - got (Variable), but expected one of:

• (float value)
  didn't match because some of the arguments have invalid types: (Variable)
• (torch.FloatTensor other)
  didn't match because some of the arguments have invalid types: (Variable)
• (float value, torch.FloatTensor other)

[qfettes]

Should be something like:

u = expected_dist.t().unsqueeze(-1) - dist
loss = self.huber(u) * (self.tau.view(1, -1) - (u.detach() < 0).float()).abs()
loss = loss.mean(1).sum()

[angmc]

When I last looked at this it ran after converting to a variable:
u=expected_quant-dist
huber_loss = 0.5 * u.abs().clamp(min=0.0, max=k).pow(2)
huber_loss += k * (u.abs() - u.abs().clamp(min=0.0, max=k))
quantile_loss = (autograd.Variable(tau.cuda()) - ((u < 0).float())).abs() * (huber_loss)
loss = (quantile_loss.sum() / num_quant)

[LRiver-wut]

Friend, this a question.

[LRiver-wut]

It confused me.