Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lua-resty-waf breaks nginx even in simulate mode #2378

Closed
dcherniv opened this issue Apr 19, 2018 · 6 comments
Closed

lua-resty-waf breaks nginx even in simulate mode #2378

dcherniv opened this issue Apr 19, 2018 · 6 comments

Comments

@dcherniv
Copy link

NGINX Ingress controller version:
0.13.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:59:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.2-gke.1", GitCommit:"4ce7af72d8d343ea2f7680348852db641ff573af", GitTreeState:"clean", BuildDate:"2018-01-31T22:30:55Z", GoVersion:"go1.9.2b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:
GCP

  • Cloud provider or hardware configuration:
    GCP
  • OS (e.g. from /etc/os-release):
    container os
  • Kernel (e.g. uname -a):
    `Linux nginx-ingress-controller-5d6959bbb8-drq8x 4.4.86+ Basic structure  #1 SMP Thu Dec 7 20:11:11 PST 2017 x86_64 GNU/Linux

What happened:
Lua-resty-waf causes nginx to return 403 for requests that were previously working in 0.10.2 even in simulate mode. When this happens, lua doesn't log anything to the console. I just see nginx returning 403s.

What you expected to happen:
Lua notifying but not blocking traffic, nginx passing traffic through when lua-resty-waf is in simulate mode.

How to reproduce it (as minimally and precisely as possible):
Update nginx controller to 0.13.0 add lua annotations:

nginx.ingress.kubernetes.io/lua-resty-waf: "simulate"
nginx.ingress.kubernetes.io/lua-resty-waf-ignore-rulesets: "41000_sqli, 42000_xss"

Try to make a semi-complex POST request, nginx returns:

2018-04-19 11:52:56.000 EDT
REDACTED - [REDACTED] - - [19/Apr/2018:15:52:56 +0000] "POST /REDACTED/get HTTP/1.1" 403 170 "-" "Apache-HttpClient/4.5 (Java/1.8.0_161)" 493 0.000 [REDACTED-service-80] - - - -

I don't see a usual lua log entry that would have followed if it had been caught by lua. I.E. the following is not present when i get 403s:

2018/04/19 15:48:51 [warn] 615#615: *3024 [lua] log.lua:52: {"timestamp":1524152931,"method":"GET","uri":"REDACTED/healthz","id":"15b935cd5521fa75c93d","client":"REDACTED","alerts":[{"match":10,"msg":"HTTP Response Splitting","id":40010},{"match":32,"msg":"HTTP Response Splitting","id":40011},{"logdata":8,"match":8,"msg":"Request score greater than score threshold","id":99001},{"logdata":8,"match":8,"msg":"Request score greater than score threshold","id":99002},{"logdata":8,"match":8,"msg":"Request score greater than score threshold","id":99003}]} while logging request, client: REDACTED, server: REDACTED, request: "GET /REDACTED/healthz?advanced%0d%0aContent-Length:%200%0d%0a%0d%0aHTTP/1.1%20200%20OK%0d%0aContent-Type:%20text/html%0d%0aContent-Length:%2035%0d%0a%0d%0a<html>Sorry,%20System%20Down</html> HTTP/1.1", upstream: "http://10.56.11.104:8080/cache/cache-service/healthz?advanced%0d%0aContent-Length:%200%0d%0a%0d%0aHTTP/1.1%20200%20OK%0d%0aContent-Type:%20text/html%0d%0aContent-Length:%2035%0d%0a%0d%0a<html>Sorry,%20System%20Down</html>", host: "REDACTED"

Anything else we need to know:
If i downgrade back to 0.10.2, disable lua annotations and upgrade back to 0.13.0, the same exact request works fine.

REDACTED- [REDACTED] - - [19/Apr/2018:15:58:32 +0000] "POST /REDACTED/get HTTP/1.1" 200 89 "-" "Apache-HttpClient/4.5 (Java/1.8.0_161)" 540 0.009 [REDACTED-80] REDACTED:8080 61 0.009 200
@ElvinEfendi
Copy link
Member

Hi @dcherniv, can you give an example POST request that I can use to regenerate this myself? Also please post the generated Nginx configuration.

@dcherniv
Copy link
Author

@ElvinEfendi

curl -X POST --header "Content-Type: application/json" --header "Accept: application/json" -d "{  
  \"sessionID\": \"BLAHBLAH\"                 
}" "https://REDACTED:443/cache/get"

I can't reproduce on my test cluster. The issue only manifests when i promote the controller to our dev environment.

Will get back to this ticket with more info.
Another important bit of info we use squid for outgoing calls to the controller.

@dcherniv
Copy link
Author

@ElvinEfendi
Some more information that might be helpful. The below is from my resulting config file. It seems that SIMULATE option only applied to access_by_lua_block.
And it seems that all of the rest of the filters dont have waf:set_option("event_log_level", ngx.WARN)
I'm new to lua-resty-waf but wouldn't the below configuration produce the exact effect that i observed?
That is the request was dropped by either header or body filters and lua didn't log anything because waf:set_option("event_log_level", ngx.WARN) was not set.

                      access_by_lua_block {
                                local lua_resty_waf = require("resty.waf")
                                local waf = lua_resty_waf:new()

                                waf:set_option("mode", "SIMULATE")
                                waf:set_option("storage_zone", "waf_storage")
                        waf:set_option("allowed_content_types", { "text/html", "text/json", "application/json" })
                                waf:set_option("event_log_level", ngx.WARN)

                                waf:set_option("ignore_ruleset", "41000_sqli")

                                waf:set_option("ignore_ruleset", "42000_xss")

                                waf:exec()
                        }
                        header_filter_by_lua_block {
                                local lua_resty_waf = require "resty.waf"
                                local waf = lua_resty_waf:new()
                                waf:exec()
                        }
                        body_filter_by_lua_block {
                                local lua_resty_waf = require "resty.waf"
                                local waf = lua_resty_waf:new()
                                waf:exec()
                        }

                        log_by_lua_block {

                                local lua_resty_waf = require "resty.waf"
                                local waf = lua_resty_waf:new()
                                waf:exec()

                        }

@ElvinEfendi
Copy link
Member

ElvinEfendi commented Apr 20, 2018 •
edited
Loading

I'm new to lua-resty-waf but wouldn't the below configuration produce the exact effect that i observed?

@dcherniv in the example at https://github.com/p0pr0ck5/lua-resty-waf#synopsis it explicitly mentions that options will be inherited. Also the default mode is SIMULATE(https://github.com/p0pr0ck5/lua-resty-waf#mode) so even if inheritance does not work it should still be running in simulate mode in every phase and not block requests.

@ElvinEfendi
Copy link
Member

@dcherniv at this point I'd try setting error-log-level to debug(https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/configmap.md#error-log-level) and see if there's anything standing out in the logs.

@dcherniv dcherniv reopened this Apr 20, 2018
@dcherniv
Copy link
Author

I will try to reproduce tomorrow via Postman, maybe it's doing something that i cannot reproduce with curl.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ElvinEfendi @dcherniv