Ask HN: What do you look for in compliance reporting tools?

Added context: I've been working on a security scanning tool called PenZen (https://penzen.app) ad interested in how others have solved it.

I worked in this space as a "virtual" CISO for a friend's company for several years (2020–2023) before giving up in frustration, so take the following with that caution.

Automated report generators made for very pretty graphics for the C level executives but failed us on actionable items for the managers and staff who had to actually fix exposures.

There was no sanity checking on some of the layout tools used, e.g. a crazy long endpoint URL would either spill off the side of the page (and be truncated) or be truncated with an ellipsis. In neither case could we tell what the original URL was without tracking down the scan log and searching it line by line.

If you're going to allow me to add my branding to the report I've paid to generate, don't charge me an arm, a leg, and an unmentionable part of anatomy to do so. And prefer SVG graphics so you can scale up/down/sideways as necessary. Or document the dimensions of the PNG/JPG you need. Or both.

One tool we used had two report options: executive level, which was a 10–15 page high level summary that was barely useful to brief senior management, or core dump, which was a potentially hundreds of pages long detailed report on each and every end point hit by the scan. There needed to be something in between (e.g. filter by severity or compliance area or…something). Generate multiple content types (pdf, pptx, html, xml, md, JSON, etc).

As you ask…there’s multiple audiences. Reports need to be generated for each. And the act of generating the report itself can be a compliance event, so the options selected or ignored for generating the report themselves may need to be recorded somewhere for a future auditor or litigator.

We typically used the service's automated process to do a scan and generate the initial report, and then manually edit the report to make it more appropriate for the given audience we were presenting to.

One service we used repeatedly until they priced themselves out of viability for us had multiple API endpoints for different types of scans where each API had wildly different parameters for the request (e.g. ?domain=example.com in one request would be ?ddns=example.com in another) and the results would have variably different JSON responses that had to be manually inspected almost every time.

There was no concept of a “organizational” account on a number of the services we tried. If I requested a scan under my account, my partner could not access the results unless he logged in as me. It would not have mattered if we were vCISO consultants or a corporation attempting to use the service.

Scan results were not directly actionable. If you're going to tell me something may be exposed to a particular CVE, link the CVE, and if possible what tripped the CVE.

As I said, I don't do this any more, and have zero interest in returning to it.

It was frustrating to be pinched between customers who wanted to spend as little money as possible checking off the “we are so secure” boxes and the service providers viewing the market as a flood of cash to grab as quickly as possible. It made zero sense for us to build our own tools out, yet once we got out of the trial phase of many of the tools the increased cost was beyond what we could bill our own customers.