Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Way simpler recognition #15

Open
zoeyfyi opened this issue Mar 13, 2017 · 51 comments
Open

Way simpler recognition #15

zoeyfyi opened this issue Mar 13, 2017 · 51 comments

Comments

@zoeyfyi
Copy link
Collaborator

zoeyfyi commented Mar 13, 2017

Turns out this is all you need to get the 128 dim vector

# Convert to RGB
print "Converting frame to RGB"
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

# Align face
print "Aligning face"
alignedFace = self.aligner.align(96, frame, skipMulti=True) # skip image if more than one face is detected

# Extract features
if alignedFace is not None:
    print "Initializing neural net"
    with openface.TorchNeuralNet(model="./data/nn4.small2.v1.t7") as net:
        print "Starting forward pass"
        features = net.forward(alignedFace)

        print features
else:
    print "No face found"

Now I just need to put it through the cascade....
@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

That's it?
And it works?

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

Yep, 1 fps but it works!

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

Is there time to dump it in a thread as we had before?

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

Maybe... im thinking of rewriting a much simpler version with this new code. Instead of mapping people to names, we can give then a color so their is no need for training

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

All you need to do is find the difference between the two arrays and it give the singularity of those two frames!

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

Which is technically face recognition, and a lot faster to explain than retraining the network if some wants to see it work

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

I have just pushed a working copy on master, as long as it doesn't loose detection, the bounding box around a face should remain the same colour. If you can could you test it with your webcam?

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

I will do, just need to make sure the dependencies work.

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017 •
edited
Loading

What did we have to do to use the correct manifest?

I was also wondering, in the README we have

sudo apt-get install luarocks for NAME in dpnn nn optim optnet csvigo cutorch cunn fblualib torchx tds; do luarocks install $NAME; done

As torch includes its own luarocks, are we overwriting the manifest here?

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

Dependencies are installed. Just need to fix torch.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017 via email

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

The dependencies are installed, just getting " 'th' no such file or directory'" when net.forward is called.

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

Just needed to export th to $PATH, it is drawing a box now.

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

And it seems to be a consistent colour

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

Fantastic. If you checkout the test branch I have made a program to help amber demo it better, could you check that works fine. Press keys 1-4 for the diffrent modes.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

One sec, still pushing...

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

OK, I will have a look. Moving the call to create the net outside of the loop improved performance slightly, but I still have a ~2.5-3 second latency between making a motion and it showing on screen.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

Well this commits so large its not pushing...

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

What did you change?

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

This is what im going to send Amber:

In the new program you press the numbers 1 to 4 to switch between the different parts of face recognition. If you need to explain how the program works you can do the following:

Press "1"
This is the bounding box mode. In this mode we draw a box around all the faces in the frame. To do this look at a 16x16 block of pixels and find its gradient (which way its pointing) to generate a HOG image. With this data we compare it against a HOG face pattern which is the combination of 100s of 1000s of HOG images to see if each part of the image is a face. Once we find a face we draw a box around it.

Press "2"
With the position of the face we use the same HOG image to draw 68 points on every face called landmarks to help us with the next step.

Press "3"
Usings the face landmarks we rotate, scale and shear the image so the face is fairly central, this helps to make the neural network more accurate since the faces look more simular.

Press "4"
We pass this data through a neural network which has be pre-trained on almost 1 million photos. For every face we get a 128 int vector which describes all the features which make up your face. Now to recognise the faces between each frame we simply look at the faces from last frame which were most similar and match them up.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

The problem is to ensure none of the data is missing I had to copy the data files outside of the data directory to ensure the git clone actually pulls them down.

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

I'm not sure if Amber will have time to clone it.

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

We could put it somewhere else online without the videos, as she already has those.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

Now re cloning and pushing without the videos... wow this repos large

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

After the 100th push, it finally went through!

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

so long as nothing in the data folder changes, the clone script should work fine to pull the last commit from the test branch and then copy the files across.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

They only files it needs are dlib_shape and nn4.small2.v1.t7 which are now in the root directory so it wont fail if the copy fails

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

Good, the copy just dumps the data folder from our backup into /FaceDetector.

If the copy fails, the videos won't work, but that should be it.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

I rewrote the whole thing (apart from webcam.py) but I had to remove the videos to push it, so can you tell me if everything still works?

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

If you look at FaceDetector.py those 134 lines implements almost all the features we had before! If only we had looked at openface's docs sooner

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017 •
edited
Loading

It actually took ~2 seconds to checkout that commit

  • Mode 1 shows random colours
  • Mode 2 shows landmarks
  • Mode 3 shows the aligned face, but the aligned face includes the bounding box
  • Mode 4 shows a continuous colour, very slowly

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

Fantastic!

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

Do we want to add this as a separate program from the main one?

This works, but to a judge it will look much less interesting than the full UI.

I'm going to see how it works as a filter.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

I was thinking of just adding the tool bar to this one. The only thing this doesnt do is counting line.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017

Since we can split impostors so easily, the spy network, with realtime training, can become a reality now!

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

I just pushed to the test2 branch with a filter containing the detection stuff.

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

If the test2 branch works for you I can update the camera number and merge it into test, unless there is a more functional version.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017 via email

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

That was what I tried to do. The whole recognition setup is just a filter now, and the keybindings still work.

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

Primarily I think the old stuff is needed just because she needs to demo it and the new stuff is really slow, and will be even slower on the laptop.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017 via email

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017 via email

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

CPU usage for just the open face stuff was about 5% for me, getting ~5 FPS on just the bounding box and ~1 FPS on the recognition.

The performance didn't change for me on the filter implementation.

How do the two branches perform on your system?

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017 via email

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

test2 uses the same files as the other branch.

We could just have Amber clone each of them, the command is just

git clone [repo] -b [branch] -depth=1

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 13, 2017 via email

@tpb1908
Copy link
Owner

tpb1908 commented Mar 13, 2017

Sounds good. I assume that the place she is staying has internet access.

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 14, 2017 via email

@tpb1908
Copy link
Owner

tpb1908 commented Mar 14, 2017 via email

@zoeyfyi
Copy link
Collaborator Author

zoeyfyi commented Mar 14, 2017

Sounds like she cloned it without an Internet connection which cleared out the directory.

@tpb1908
Copy link
Owner

tpb1908 commented Mar 14, 2017

OK, the restore script should fix that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zoeyfyi @tpb1908