Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rc-cts-urlscanio Not Returning Hits #69

Open
lmahoney1 opened this issue Mar 8, 2021 · 3 comments · May be fixed by #70
Open

rc-cts-urlscanio Not Returning Hits #69

lmahoney1 opened this issue Mar 8, 2021 · 3 comments · May be fixed by #70

Comments

@lmahoney1
Copy link

Description

Our analysts mentioned that they had not seen a hit from the urlscan.io cts in awhile. We deployed rc-cts-urlscanio from the community.

It appears that the check the CTS does to determine if the results are malicious or not no longer works (around line 136). Maybe URL Scan.io changed their API response?

We used the following url https://urlscan.io/result/38135c02-509d-471f-91a7-0466060c3474/ for testing. We found that the urlscan.io GUI marked the URL as 'potentially malicious', however when submitting it to a running CTS we got an empty hit.

The following code sample may explain better:

import requests
headers = { 'Content-Type': 'application/json' }
# looking up URL in urlscanio
url_results = requests.get('https://urlscan.io/api/v1/search/?q=page.url:"router-ed1c9cd2-b691-4867-b4d4-fb8843fa91b1.eastus.cloudapp.azure.com"', headers)

# the first search result found - one of the times the URL was analyzed on the site
result = url_results.json().get("results")[0]
# ID of the result / analysis
result_id = result.get("_id")

# getting a detailed report of the analysis
result_response = requests.get('https://urlscan.io/api/v1/result/38135c02-509d-471f-91a7-0466060c3474', headers).json()

# the current 'is it malicious' check
result_response.get("stats").get("malicious") # outputs 0 -> not malicious, no hit is returned

# potential new 'is it malicious' check
result_response.get('verdicts').get('urlscan').get('malicious') # outputs True, a hit should be generated

In the example above I'm only checking one of the two (at the time of writing this) analyses available for the URL. This is just for the example, I verified that both of the analysis have the same issue.

Describe How to Reproduce

Submit a 'potentially malicious' URL to the rc-cts-urlscanio CTS.
@lmahoney1 lmahoney1 linked a pull request Mar 8, 2021 that will close this issue
Open
4 tasks
@lmahoney1
Copy link
Author

Also, I tried spinning this up in apphost after adding all of the required apphost files, but I wasn't able to communicate with it. Maybe I was doing it wrong?

I uncommented the EXPOSE 9000 line in the Dockerfile I used to generate my docker image. The problem was communicating with the pod/container the app runs on from outside the apphost server. I tried sending requests to both the app host server's IP as well as the container ID (retrieved from sudo kubectl describe pod <rc-cts-urlscanio pod ID>.

@mscherfling
Copy link
Collaborator

Custom Threat Feeds cannot be deployed inside a container. This is because containers don't have fixed IP Addresses which the webhook requires when registering it with Resilient.

@lmahoney1
Copy link
Author

That's too bad but it makes sense. The following line in the Dockerfile gave me hope

# uncomment to expose port only if a custom threat feed
#EXPOSE 9000

@lmahoney1 lmahoney1 mentioned this issue Apr 20, 2021
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rc-cts-urlscanio-1.0.1 lmahoney1/resilient-community-apps
2 participants
@mscherfling @lmahoney1