Adds helm operator that provides a controller that handles

• RapiDAST - main CRD that manages running RapiDAST as a job on the cluster
• ~~RapiDASTCC - provides PVC that is used by RapiDAST for persistence, along with a pod using the PVC for simple copying of artifacts produced by the jobs~~

This operator code has been refactored to account for API_KEY option now as an environment variable. In the RapiDAST CRD, the boolean option for apiKeyRequired can be set to true. As implemented, this will create a secret, and the api key will have to be updated there manually.

Feedback welcome.

• Edit - Moved beyond needing the RapiDASTCC API. PVC now created as needed when creating RapiDASTs.

Hi,

I got a Mac M1 recently and I am trying to run the zaproxy tests against a config by placing the open api url and target url as expected in the config/config.yml and then following steps below but getting the below error.

Any help would be appreciated, thanks.

rapidast (master %) $ docker-compose up zaproxy
[+] Running 1/0
 ⠿ Container zaproxy  Created                                                                                                                                                                                                                                              0.1s
Attaching to zaproxy

Trying to run the scan as per the README file:

$ docker-compose exec zaproxy python /zap/scripts/apis_scan.py reports                                                                                                                                        1
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x4002245400>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8090): Max retries exceeded with url: http://zap/JSON/core/action/newSession/?apikey=cnmeemn7jp7ijd8rl5u14q40v8&name=%2Fzap%2Fresults%2Freports%2Fsessions%2F20220419-170600%2Fsession1&overwrite=True (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4002245400>: Failed to establish a new connection: [Errno 111] Connection refused')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/zap/scripts/apis_scan.py", line 313, in <module>
    create_session(session_fullpath_name)
  File "/zap/scripts/apis_scan.py", line 17, in create_session
    zap.core.new_session(name=session_name, overwrite=True)
  File "/usr/local/lib/python3.8/dist-packages/zapv2/core.py", line 357, in new_session
    return six.next(six.itervalues(self.zap._request(self.zap.base + 'core/action/newSession/', params)))
  File "/usr/local/lib/python3.8/dist-packages/zapv2/__init__.py", line 200, in _request
    data = self._request_api(url, get)
  File "/usr/local/lib/python3.8/dist-packages/zapv2/__init__.py", line 180, in _request_api
    response = self.session.get(url, params=query, proxies=self.__proxies, verify=False)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 513, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPConnectionPool(host='127.0.0.1', port=8090): Max retries exceeded with url: http://zap/JSON/core/action/newSession/?apikey=cnmeemn7jp7ijd8rl5u14q40v8&name=%2Fzap%2Fresults%2Freports%2Fsessions%2F20220419-170600%2Fsession1&overwrite=True (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4002245400>: Failed to establish a new connection: [Errno 111] Connection refused')))

Adding the ability to scan from URLs specified in a URL Scan config file that does not have an OAS definition. Note that the reason the importurls method has been added here is because the zap python API project currently does not support exim (the new ZAP addon for importing urls) and only makes use of the deprecated importurls addon. I've added support for exim imports instead in the new method.

Podman 3.1.0 and above support the "U" flag to change ownership of
volumes.
This makes the `unshare chown` unecessary when using podman >= 3.1
IIUC, it does not affect older versions, which will simply ignore the
unknown flag.

List of changes:

• API_KEY was removed from config.yaml and moved to an env file
• entrypoint.sh and entrypoint_ui.sh files created
• config/requirements.txt was created
• tty: true and stdin_open: true removed from compose files as they have no effect

based on https://github.com/RedHatProductSecurity/rapidast/pull/14 so changes should be more easy to see once that PR is merged.

Added rapidast-scan action as example

The main reason for the wrapper is to avoid the need to run unshare, by formatting a defined user mapping, such that the zap user maps the host user. This way ZAP can write in the ./results share without needing to change ownership.

source: https://github.com/containers/podman/blob/main/troubleshooting.md#39-podman-run-fails-with-error-unrecognized-namespace-mode-keep-iduid1000gid1000-passed In very recent podman version, this hack can be simplified.

This change is backward compatible : nothing prevents the user to use any old methods for starting RapiDAST.

Note: Unlike runenv.sh, this script does not attempt to stop the container in case it is already running. The reasoning is : if RapiDAST is currently undergoing a looooong scan, a user probably prefers the new command to fail, rather than cancelling the current scan. However, we could improve that.

Other minor changes: In README.md:

• removed the reference to [podman|docker] pull, as it refers to a different image, and afaik, will be done by the compose command
• updated with the new command, removed references to unshare
• few minor readability/consistency update

In runenv.sh / runenv-ui.sh: echo "deprecated" message

• Adding a docker version for starting the zaproxy tool
• Adding a parameterized http authentication script to be able to pass the parameters from the config where it wont be pushed to any repo accidentally

For openshift console runs:

HttpSenderScriptFilePath: 'scripts/add-token-cookie-param.js'
HttpSenderScriptDescription: 'add a cookie to each HTTP request'
HTTPParams: {"cookieName": 'openshift-session-token', "cookieVal": 'sha256~**'}

Some code was changed manually. So it needs some testing before merging. Please, review changes in python files carefully.

During my work on the pre-commit fixes I found that some checks are not so convenient to use or they edit files which they shouldn't touch. Thus the pre-commit config was updated.

The pydocstyle checks are still missing here:

scripts/config.py:1 at module level:
        D100: Missing docstring in public module
scripts/gen_zap_script/lib.py:1 at module level:
        D100: Missing docstring in public module
scripts/gen_zap_script/lib.py:30 in private class `Script`:
        D205: 1 blank line required between summary line and description (found 0)
scripts/gen_zap_script/lib.py:30 in private class `Script`:
        D212: Multi-line docstring summary should start at the first line
scripts/gen_zap_script/lib.py:87 in public function `add_and_load_script`:
        D103: Missing docstring in public function
scripts/gen_zap_script/lib.py:114 in public function `delete_all_loaded_scripts`:
        D103: Missing docstring in public function
scripts/apis_scan.py:1 at module level:
        D100: Missing docstring in public module
scripts/apis_scan.py:18 in public function `create_session`:
        D103: Missing docstring in public function
scripts/apis_scan.py:38 in public function `enable_httpsender_script`:
        D103: Missing docstring in public function
scripts/apis_scan.py:61 in public function `create_context`:
        D103: Missing docstring in public function
scripts/apis_scan.py:158 in public function `enable_passive_scanner`:
        D103: Missing docstring in public function
scripts/apis_scan.py:164 in public function `import_urls`:
        D205: 1 blank line required between summary line and description (found 0)
scripts/apis_scan.py:164 in public function `import_urls`:
        D212: Multi-line docstring summary should start at the first line
scripts/apis_scan.py:171 in public function `get_apis`:
        D103: Missing docstring in public function
scripts/apis_scan.py:225 in public function `check_scan_id`:
        D103: Missing docstring in public function
scripts/apis_scan.py:232 in public function `start_active_scanner`:
        D103: Missing docstring in public function
scripts/apis_scan.py:295 in public function `start_spider`:
        D103: Missing docstring in public function
scripts/apis_scan.py:316 in public function `wait_for_passive_scanner`:
        D103: Missing docstring in public function
scripts/apis_scan.py:326 in public function `generate_report`:
        D103: Missing docstring in public function
scripts/gen_zap_script/lib_usage_example.py:1 at module level:
        D100: Missing docstring in public module
scripts/gen_zap_script/lib_usage_example.py:6 in public function `js_passive_script_example`:
        D103: Missing docstring in public function
scripts/gen_zap_script/lib_usage_example.py:22 in public function `js_active_script_example`:
        D103: Missing docstring in public function
scripts/gen_zap_script/cli.py:1 at module level:
        D100: Missing docstring in public module
scripts/gen_zap_script/cli.py:57 in public function `add_finding_group`:
        D103: Missing docstring in public function
scripts/gen_zap_script/cli.py:101 in public function `ms_check`:
        D103: Missing docstring in public function
scripts/gen_zap_script/cli.py:163 in public function `file_lines_or_default`:
        D103: Missing docstring in public function

Looking for help with it.

UPDATE: We decided to disable the docstring check and split from this task

The results directory should not have the "U" unshare mode. The reason is that the ./test/scan-example-with-podman.sh script intends to create a child directory in results. "U" here removes ownership from the EUID too early, and it's not the right directory to unshare anyway. We want to unshare the child. This patch reverts the addition of "U" mode on the results mount.

FYI, I just noticed that there is a hardcoded API key on this line:

https://github.com/RedHatProductSecurity/rapidast/blob/3e477d72cd5c1fb721a7775769def4d988926e6d/entrypoint.sh#L12

/cc @jpweiser since it was your commit, you may care.

Added model and a query object. It can work with config data like this:

Config:

general:
  serviceName: 'demo1'
  resultDir: '/results/'
  # appDir: '/zap'
  localProxy: 
    http: "http://127.0.0.1:8090"

The query:

>>> config = Config(config_file=args.rapidast_config)
>>> config.q.general.localProxy.http.value
'http://127.0.0.1:8090'

Also it supports another way to call it:

>>> config.q["general"]["localProxy"]["http"].value
'http://127.0.0.1:8090'

(Please, see tests for additional examples)

If there is no a key in the config, the chain won't fail with a python error, it will return None instead.

>>> config.q.general.WRONG_KEY.localProxy.http.value
None

It will save us from ugly chains like:

config.get("general", {}).get("localProxy", {}).get("http")

and it will safe us from unexpected python errors if a key is not presented in the config file

P.S. the PR is in progress, it needs additional work before final review