Automating Authenticated Vulnerability Scanning of GCP Hosts with Rapid7’s InsightVM

…and I’m not just talking about run a scheduled scan with credentials already set. I’m talking about deploying a virtual scanner appliance into a subnetwork of a project in GCP (provided that you have the correct Organizational level role to do so), performing an ad-hoc scan against hosts (complete with authenticated from keys generated on the fly), uploading report, then terminating the scanner (and the keys). And being able to do it again in another subnetwork inside another project. The whole process took about 30 minutes to scan 3 hosts and upload a report to a Google Storage Bucket.

I worked on this fun project to:

1. Keep practicing Go (just started a couple months ago and loving it!) and getting more familiar with GCP APIs
2. As a PoC to be able to scan GCP hosts in a network without having to add agents or deal with managing/rotating SSH keys/Auth Failures, but still get reports.

P.S. Github link at the bottom

So! The workflow:

1. Client request linux hosts in project → network → subnetwork to be scanned. You (or someone with org level permissions) approve/launch or deny request.
2. If approved (aka You again or me in this case), an image of pre-configured scanner is taken from scanner image project and an instance is created inside of client’s subnetwork.
3. Script is slept for a few minutes for instance to start and get an IP address and the scan engine to start. Once it has an IP, the scanner’s internal IP is added to a blacklist so the script doesn’t waste time adding a private key to it and the scanner scanning itself.

Here are the hosts to be scanned:

1. As InsightVM is starting up, it will redirect to a starting.html page shown below. This means that you can’t log into the scanner just yet. So I created a waiter that checks for the login page every minute and if the request gets redirected to the starting.html page, that means the scanner isn’t ready. When the redirecting stops, the script can move forward with setting up the scan as it can then reach the login.jsp page and authenticate with the API.

starting.html

login.jsp, scanner is ready

5. A firewall rule is created to allow all 0–65535 ports ingress from the internal IP address of the scanner to all hosts in the network.

6. A request is made to GCP Secret Manager (if you are on GCP and haven’t check this out, do so!) to get the admin credentials of the InsightVM scanner to use the API. Basic authentication is used to communicate with the scanner, the credentials must be in this format (using Python to explain):

import base64
auth_string = "username:password"
base64.b64encode(auth_string)
# header = {"Authorization”: f“Basic {auth_string}"}

So when making a GET or POST request, for the Authorization header, the script uses “Authorization”: “Basic <auth_string>”.

7. A private and public key are created. This is generating the keys in memory which will be gone once the instance is terminated. This also makes the keys unique every time this script is called.

Public key is then formatted in GCP‘s required format (using Python and Fstrings).

username = "bot"
ssh-pubic-key = "ssh-rsa AAA..." #insert full-key
# when using console to add ssh-key to instance
add_via_console_format = f"{key} {username}"
# when using API to add ssh-key to instance
add_via_API_format = f"{username}:{key} {username}"

8. The script then creates a list of all the instances in the network that are running minus the ones in the blacklist. It then adds the public key to the metadata of the instance.

Host without keys:

Host with keys:

9. The private key is added to InsightVM and a new site is created. Think of a site like a business unit or folder that assets are added to and it keeps a history of all scans run against those assets and any credentials needed to authenticate. The hosts, from step 8, are added to the site and a scan is initiated.

Here is a screenshot of the host added:

Here is the private key added by the script:

This is how the scanner will authenticate to the instances.

10. A waiter is started to check on the status of the scan every minute until its status is finished. When the scan is finished, a report is created and uploaded to a private bucket to be reviewed.

Completed Scan:

Here is the assets after the scan has finished:

See those sweet successful authentications??

Here is the automated report sitting pretty in my bucket.

11. And the last step… everything is cleaned up. The scanner is terminated, the keys are removed from the hosts that were scanned, and the firewall rule is deleted. Like it was never… there…

Oh and here is a sample of the report:

Final thoughts:

• Why InsightVM vs other scanners? Well it was just easier to get a trial key. It’s possible to do this PoC for Nessus if you have can use the Tenable.io API (not just the Nessus scanner itself since you cannot add credentials via API). I’m not completely sure if you can do this with Qualys given their virtual scanners are deployed through GCP Deployment Manager which makes it a bit more complicated.
• Why not Docker? Baby Steps! I wanted to see how possible this was first.
• One big flaw is the scanner by default will be launched in the central region in the subnetwork, with some tweaking of the code, this can be more dynamic.
• Anyway, here is the code for it (I just started using Go around Christmas 2019 so go easy on me fellow Gophers, I know there are some improvements needed). Also, remember, only scan what you have permissions to scan. This is just the code, you will still need to set up the InsightVM scanner image.

Anywho, thanks for taking the time to check out my post with another weird project. Onto the next one! :)