bad performance of pytorch model compared to lua

[shuuchen]

Hi,

I compared the model performance of both pytorch and lua.
Pytorch model is trained 30 epochs while lua model is provided by the author.
The same training data is used.

However, when I test the following image, 33 walls were detected by lua model while only 11 were detected by pytorch model.

The picture is

lua output:
(in which 33 walls are detected)

222	121	222	144	door	1	1	

128	161	128	210	door	1	1	

232	77	232	102	door	1	1	

155	96	155	143	door	1	1	

186	68	186	83	door	1	1	

41	67	41	115	door	1	1	

45	48	97	48	door	1	1	

242	48	259	48	door	1	1	

46	227	96	227	door	1	1	

159	48	178	48	door	1	1	

17	145	40	145	door	1	1	

59	145	110	145	door	1	1	

159	147	181	147	door	1	1	

141	227	190	227	door	1	1	

188	50	201	61	washing_basin	1	1	

212	50	229	66	washing_basin	2	1	

203	173	220	226	cooking_counter	1	1	

234	51	273	73	bathtub	1	1	

200	121	220	145	entrance	1	1	

164	52	175	71	toilet	1	1	

2	71.5	52	91.5	closet	1	1	

73	85.5	123	105.5	bedroom	1	1	

145	60	195	80	restroom	1	1	

188.14285714286	82.247619047771	238.14285714286	102.24761904777	washing_room	1	1	

166.52380952371	113.07619047614	216.52380952371	133.07619047614	corridor	1	1	

227.5	65.5	277.5	85.5	bathroom	1	1	

2	120	52	140	closet	1	1	

46	175.5	96	195.5	bedroom	1	1	

148.54046242775	177.54046242775	198.54046242775	197.54046242775	kitchen	1	1	

203	117.666666667	222	117.666666667	wall	1	1	

273.5	47.8666666667	273.5	104	wall	1	1	

232	104	273.5	104	wall	1	1	

232	104	232	117.666666667	wall	1	1	

222	117.666666667	232	117.666666667	wall	1	1	

185.5	47.8666666667	185.5	93	wall	1	1	

155.333333333	93	185.5	93	wall	1	1	

222	156.5	222	227.333333333	wall	1	1	

128	227.333333333	222	227.333333333	wall	1	1	

197.5	156.5	222	156.5	wall	1	1	

197.5	146.733333333	197.5	156.5	wall	1	1	

14	227.333333333	128	227.333333333	wall	1	1	

14	144.6	14	227.333333333	wall	1	1	

13.6666666667	47.6666666667	13.6666666667	116	wall	1	1	

13.6666666667	47.6666666667	41.3333333333	47.6666666667	wall	1	1	

232	47.8666666667	273.5	47.8666666667	wall	1	1	

222	117.666666667	222	146.733333333	wall	1	1	

185.5	47.8666666667	232	47.8666666667	wall	1	1	

155.333333333	47.8666666667	185.5	47.8666666667	wall	1	1	

232	47.8666666667	232	104	wall	1	1	

41.3333333333	47.8666666667	155.333333333	47.8666666667	wall	1	1	

41.3333333333	47.8666666667	41.3333333333	116	wall	1	1	

128	144.6	155.333333333	144.6	wall	1	1	

128	144.6	128	227.333333333	wall	1	1	

41.3333333333	144.6	128	144.6	wall	1	1	

155.333333333	47.8666666667	155.333333333	93	wall	1	1	

197.5	146.733333333	222	146.733333333	wall	1	1	

155.333333333	146.733333333	197.5	146.733333333	wall	1	1	

222	146.733333333	222	156.5	wall	1	1	

41.3333333333	116	41.3333333333	144.6	wall	1	1	

13.6666666667	116	41.3333333333	116	wall	1	1	

155.333333333	93	155.333333333	146.733333333	wall	1	1	

13.6666666667	144.6	41.3333333333	144.6	wall	1	1	

13.6666666667	116	13.6666666667	144.6	wall	1	1	

pytorch output:
(in which only 11 walls are detected)

256     256
11
12.449275362318842      208.3768115942029       116.59958071278825      208.3768115942029       3       0
37.315602836879435      107.09478021978023      37.315602836879435      135.16484142914493      0       6
203.3   135.16484142914493      203.3   208.3768115942029       0       2
116.59958071278825      135.16484142914493      203.3   135.16484142914493      0       2
116.59958071278825      135.16484142914493      116.59958071278825      208.3768115942029       2       3
37.315602836879435      135.16484142914493      116.59958071278825      135.16484142914493      0       3
116.59958071278825      208.3768115942029       203.3   208.3768115942029       2       0
12.449275362318842      107.09478021978023      37.315602836879435      107.09478021978023      0       6
12.449275362318842      135.16484142914493      37.315602836879435      135.16484142914493      6       3
12.449275362318842      107.09478021978023      12.449275362318842      135.16484142914493      6       0
12.449275362318842      135.16484142914493      12.449275362318842      208.3768115942029       3       0
142     135.0   166     135.0   door    1       1
15      135.0   36      135.0   door    1       1
53      135.0   102     135.0   door    1       1
116.0   147     116.0   193     door    1       1
12.0    161     12.0    179     door    1       1
42      208.0   88      208.0   door    1       1
130     208.0   173     208.0   door    1       1
213     46      249     67      bathtub 1       1
186     159     201     207     cooking_counter 1       1
149     46      162     64      toilet  1       1
185     109     201     134     entrance        1       1
173     45      186     58      washing_basin   1       1
191     46      209     61      washing_basin   1       1

While the corners are all predicted correctly, the walls are predicted poorly.
Since the performance of pytorch model is not checked, is it a bad model or something wrong with IP ? or should I train more epoch ?

[riti1302]

Hello @shuuchen
Did you used the same code of pytorch as provided in this repository as it is or you modified it also?
I am trying to run the pytorch code but I am having trouble in setting the path to the input files.
Looking forward to your reply.

[shuuchen]

@riti1302
My version.
https://github.com/shuuchen/FloorplanTransformation/tree/master/pytorch

[riti1302]

@shuuchen Thanks for sharing your version. I am getting the same error as I was getting in @art-programmer code. Do you know how to solve this?
keyname=floorplan task=train started
Traceback (most recent call last):
File "train.py", line 175, in
main(args)
File "train.py", line 25, in main
dataset = FloorplanDataset(options, split='train_1', random=True)
File "/home/ritika/Documents/FloorplanTransformation/pytorch/datasets/floorplan_dataset.py", line 303, in init
with open(self.dataFolder + split + '.txt') as f:
IOError: [Errno 2] No such file or directory: '../data/train_1.txt'

[shuuchen]

@riti1302

IOError: [Errno 2] No such file or directory: '../data/train_1.txt'

It says you don' t have this file.
Use '../data/train.txt' instead.

[riti1302]

@shuuchen
There is no train.txt file in data directory. Have you used the same data directory? And also can you tell me where have you put your input data?

[shuuchen]

@riti1302
try again

[riti1302]

@shuuchen The same problem persists.

[shuuchen]

@riti1302
clone the project again.
I have updated it

[riti1302]

@shuuchen Thanks. It solved my problem. Actually, I am new to pytorch and I am having trouble writing the code for prediction. It will help a lot if you send me your code for prediction.

[shuuchen]

@riti1302
what do you want to predict ?

[riti1302]

@shuuchen
If I enter a input floorplan image then a txt file (similar as one used during training) should be given out as an output.

[shuuchen]

@riti1302
I recommend you can train the model for 100 epochs

[riti1302]

@shuuchen I did that.
I want to know how you got the output (shown below) that you posted in this issue.

[shuuchen]

@riti1302
which file are you looking ? could you share the url ?

[sanwong15]

@shuuchen Hi, I also face issue training this network. I went into the Pytorch folder and follow the instruction (1: install requirment.txt, 2: python train.py -restore = 0) then it returned No such file or directory: '../data/train.txt' error. How can I solve this? Thanks

[shuuchen]

@sanwong15

you don't have the '../data/train.txt'.
just download the file, put it in the right path, and try again.

[sanwong15]

Hi. Thanks for your reply. Where can I download it? How about the test.txt file? I am not sure if I have misread the documents. Thank you for helping me out with this San

…

On Sat, Jun 22, 2019, 11:35 PM Shuchen Du ***@***.***> wrote: @sanwong15 <https://github.com/sanwong15> you don't have the '../data/train.txt'. just download the file, put it in the right path, and try again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22?email_source=notifications&email_token=ABWPLCK4ROCSFP5OLRQVWG3P3ZBAFA5CNFSM4HBMG7F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYKLZLY#issuecomment-504675503>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABWPLCO6CHB72NSJWQ37UGTP3ZBAFANCNFSM4HBMG7FQ> .

[riti1302]

@shuuchen How you test the output for a floorplan?

[shuuchen]

@riti1302
you said the walls are predicted poorly, so what model did you use ?
I want details about that

[shuuchen]

@sanwong15
my version
https://github.com/shuuchen/FloorplanTransformation/tree/master/pytorch

[riti1302]

@shuuchen As you said in this issue that you tested the image. Can you tell me what command you used to test the image?

[shuuchen]

@riti1302
Ok. That is and old issue. I have already fixed that.
Just use the following command

python train.py --task=test

[riti1302]

@shuuchen
Okay. Thanks.
And what is the use of function test_batch?

[shuuchen]

@riti1302
test_batch is for test, while test is for validation
refer to ../data and you will find the files

[riti1302]

@shuuchen Where is the output text files are stored?

[shuuchen]

@riti1302
it is in test folder.
I suggest you reading the source code as it is easy to understand.
this discussion page is too long, I recommend open a new issue in my personal page
https://github.com/shuuchen/FloorplanTransformation/tree/master/pytorch

[riti1302]

@shuuchen There is no option of creating a issue in your repository.

[shuuchen]

@riti1302
ok, all right

[riti1302]

@shuuchen I have tried training the model with 100 epochs, batch size=5 and number of images = 600. The performance of the model is very poor. Have you tried improving the accuracy? If yes, then can you tell me what changes should I make to improve the accuracy?

[shuuchen]

what train data did you use? you can use the data on my github, the result is good. 2019年6月25日(火) 14:26 Ritika Kumari <[email protected]>:

…

@shuuchen <https://github.com/shuuchen> I have tried training the model with 100 epochs, batch size=5 and number of images = 600. The performance of the model is very poor. Have you tried improving the accuracy? If yes, then can you tell me what changes should I make to improve the accuracy? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22?email_source=notifications&email_token=ADFCC2LWOS236GHPXNP6273P4GT7FA5CNFSM4HBMG7F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYPBMKY#issuecomment-505288235>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADFCC2OTAC6AMHBET7DZDI3P4GT7FANCNFSM4HBMG7FQ> .

[riti1302]

@shuuchen I used your data. Can you give me some of your training details like number of epochs, learning rate and batch size?

[shuuchen]

@riti1302
just use python train.py --restore=0
did you modify the code ?

[riti1302]

@shuuchen Yes, a little. I am saving the checkpoint every 5 epochs. Nothing else.

[riti1302]

@shuuchen Also the the default epochs is 1000. Have you trained the model on 1000 epochs?

[shuuchen]

I think 500 epoch is enough. Have you trained that much? I updated my repository. Please reload that Ritika Kumari <[email protected]>于2019年6月26日 周三下午3:41写道：

…

@shuuchen <https://github.com/shuuchen> Also the the default epochs is 1000. Have you trained the model on 1000 epochs? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22?email_source=notifications&email_token=ADFCC2PK6KVDGCZ3PPFBH5TP4MFQ5A5CNFSM4HBMG7F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYSQGIA#issuecomment-505742112>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADFCC2PA2XQ562TAJ75JTL3P4MFQ5ANCNFSM4HBMG7FQ> .

[riti1302]

@shuuchen I trained it at 100 epochs with your dataset and it is showing infeasible.

[shuuchen]

@riti1302
train more please
500 epoch is good

[riti1302]

@shuuchen I figured out my mistake. I was saving the checkpoint after every 5 epochs and the base was set to 0. When I was executing the code for validation it was still set to 0. For validation I have to change the base to 95 so that it will load the latest checkpoint. Now it's giving good accuracy. Thanks a lot for your help.

[riti1302]

@shuuchen What is the meaning of last two columns of the text file?
Also can you tell me how to detect different type of doors?

[shuuchen]

@riti1302
what do you mean by last two columns of the text file?

[riti1302]

The last two columns. For example: 1 1 after kitchen

[shuuchen]

@riti1302
Well, the last two columns are not used yet.
If you dive into the code, you can realize that.

[riti1302]

@shuuchen Okay, i got it. I am trying to train the model on my dataset but it is giving error. I posted that error above.

[shuuchen]

@riti1302
it seems you use bool object as dataloader
just check the variables according to the error messege

[atacand1920]

@shuuchen @riti1302 hello guys, can i use the task=test_batch or task=test without drn? i have a model_floorplan.t7 that i plan to use it instead of the drn dataset because i don't have an account(no access).
If you can help me with that, it will be appreciated.

[liunan123456]

@riti1302
it seems you use bool object as dataloader
just check the variables according to the error messege

@sanwong15
my version
https://github.com/shuuchen/FloorplanTransformation/tree/master/pytorch

hello @shuuchen i have traind the model for 200 epochs,but the result of the predict is very poor,can you give me a trained model to have a test? thanks,[email protected]

[1021928487]

when I run “python train.py --task=test”，output:

It's not what you run it

How can I solve this problem?@shuuchen

[joaocmd]

@shuuchen I figured out my mistake. I was saving the checkpoint after every 5 epochs and the base was set to 0. When I was executing the code for validation it was still set to 0. For validation I have to change the base to 95 so that it will load the latest checkpoint. Now it's giving good accuracy. Thanks a lot for your help.

Hello, could you explain what the change was? I do not understand what base you were referring to in your comment.

[lilu-REA]

how many GPU memory you need to train? it is huge on my side.

[BobbyZ04]

Heyyy! I wonder how did you output these combined output images?

I could only get separate output for door/line/icon/room but no combined output. and I checked IP.py and could not find anywhere they combine them together. Thank you!!