How To Tell If Your PCI Scanning Vendor Is Dangerous

Um. SSL?!

Well Rich, how would you propose a PCI ASV scan a client?  I agree with you, though— it is crazy.

You’‘ll slowly coming to realize PCI is a SCAM.  Worst of all, there’s no list of PCI-(non)compliant companies.  The only one that knows is the PCI council, and they’‘ll fine you and know one else will know.  I’‘ve been on a PCI project the past several months, and I’‘m becoming increasingly disillusioned with PCI, vendors and the security industry in general.  Hit me up some time and we’‘ll talk.

I know (intimately) of at least one ASV whose *proposals* explicitly required the customer to whitelist source addresses—and multiple sources at that. Only 1 out of, say, 1000 customers *ever* questioned it.

Oh.  Real Attackers (who don’‘t already have an inside track, which is probably much more common than you would think) use these things called proxies.  An IPS that blocks by IP is useless.  An IDS that reports by IP is doubly useless (unless you can get some HTTP request strings to identify anything, which is also unlikely if the attacker is smart).  All it really takes is to Tor to an SSL proxy.  Tor exit nodes can analyze traffic, but they can’‘t analyze SSL traffic, and the penultimate-hop SSL proxy operator only sees that Tor exit node.

A more sophisticated IPS can probably still be identified by a tool such as osstmm-afd.  Some WAFs can be identified by w3af.  A more sophisticated adversary is going to have identified most products out there, and have ways to work around them.  IDS evasion techniques go back more than ten years.  Many IDS/IPS testers claim that there are over 60 types of evasion techniques.

I am a firm believer that IPS and WAF increase the surface area of attack.  I think that it may potentially be very easy to find and execute a remote code or shell exploit on any given IPS or WAF product, given the correct set or resources to do so (which doesn’‘t have to even be the firmware or source code).

The primary problem that I see with PCI ASV assessments is that scans are often cleartext, over ISP, potentially shared (i.e. shared with other customers—even if only by misconfiguration), infrastructure.  Have you checked every fiber and Ethernet for taps?  Does your ISP’s Catalyst 6500 have a SPAN port going on your traffic?  Is your neighbor enabling it because your ISP leaked a management VLAN that had SNMP traffic on it?

Or, to put this in my best Alex Pilosov voice, "can’‘t we just BGP hijack the traffic?".

Any assessor you work with should probably be allowed to do some probing over proxies with cleartext Internet.  When the assessor wants to execute an HTTP request that potentially contains a vulnerability—this should probably be done over an SSL VPN that uses Ephemeral (EDH) keys, for perfect forward secrecy.  We already expect vulnerability assessors to make quality, snap decisions when testing our infrastructure in the case of accidental service/application crashes, concurrency loops such as deadlocks or livelocks, generation of an exorbitant amount of log data, etc.

I know that the PCI SSC requires these types of assessments, and I largely don’‘t agree with them.  To me, an external assessment like this is a waste of time.  I hate to show organizations where they are insecure.  It’s most often the case that you only find the exploits, and rarely find the root cause.

It’s like I (well, I stole this from Fortify) talk about when referring to source code reviews—and what separates application security from application penetration-testing (and yes, you can perform application penetration-testing with only source code).  Is the code obviously secure, or is it obviously insecure?  Placing vulnerabilities or exploits at the top of the security-industry-food-chain is an exercise in, what Dennis Groves refers to as, "security masturbation".

Let me go on, because this is an important topic.  Scanners, especially network vulnerability scanners, but also web application vulnerability scanners—both perform tons of tasks with network pluggable protocols and deal with a large variety of inputs.  This makes them particularly vulnerable to exploitation.  Just like IPS and WAF, scanners increase the surface attack area.  What happens if some MITM (I liked the ISP neighboring customer concept from before a lot, but the BGP one is about as good, as would be a DNS one) injects some traffic back towards the assessor’s scanning tool and owns his or her OS?  Let’s say that there is more than one set of client data available on that OS, or on that network (Qualys anyone?).

It probably takes an afternoon to setup SSLExplorer (an open-source SSL VPN server), including the forcing of EDH keys.  Most organizations move at snail pace, but when building a SOW for a PCI ASV engagement, I’‘d definitely make this a requirement in order to protect both of us.  A lot of organizations connect PCI ASVs over their already existing Extranet (or partner-networks), because many organizations already do business in this way.  The CCP has to connect somewhere, and they could also certainly be the source of a threat—so you could theoretically connect your PCI ASV there.

The PCI ASV test doesn’‘t test for ethics, so your organization is effectively running blind when hiring such an organization.  A PCI ASV is not something you want to grab out of Google Directory or the Yellow Pages.  Establishing trust is extremely important, and this often takes years of other network or application penetration-testing type work (think about starting small) before serious consideration.  This may put a lot of merchants in a corner.

However, merchants have already been forced in the corner from the ASV rule for quite some time.  When they’‘re choices were originally a full service penetration-testing company, or HackerSafe, which did many merchants pick?  Who are they still choosing?

@marcin- Give me a little credit, I slam PCI on a regular basis :)

@dre- you really need to go blog that instead of letting it languish as a comment here. For the record, I’‘ve never been overly impressed with the ASV model.

oh sure an ASV can fly someone out to your location with the scanner and scan you from inside, but are all merchants willing to pay 10 times more for that than remote scanning service?

to expand on what Ravi said above, I think that this is an example of the adage "fast, good, cheap choose two".

As far as I can see it the goal of the ASV scan, is to find as many "low hanging fuit" vulnerabilities in external facing infrastructure as possible in an automated and cheap way.  It’s not a "deep dive" into the security of systems, that’s covered in the "proper" penetration testing requirements in 11.3.

So whilst it would be possible for the ASV to go really slow and try to avoid IDS/IPS systems, that would break "cheap and fast".

I’‘d agree that white-listing of IP addresses is in itself a security vulnerability but, I’‘d suggest one that could be mitigated by the assurance that clients should be carrying out on their security vendors and some sensible controls like time-limiting the white-listing to the times of the scans’,

While I was reading the post and comments, I pretty much was thinking the same as Ravi. You get what you pay for.

There are three scans that are useful for getting an idea of how safe a system is. (I’‘m ignoring application scans here but this can be extended to applications).

1. Outside scan - tests what someone who is outside the firewall and IPS/IDS/router etc etc can see of your network.

2. Inside scan - what someone who is in your data center can see. This is a "what-if-our-firewall-fails" test.

3. Privileged test - connects to the admin console of the device (windows server ADMIN$, ssh terminal on unix, ssh on router, etc) to see what patches have been applied. This is to see what happens if a service is enabled in future or someone gets on the box.

These are all important tests to do but I certainly wouldn’‘t open up my network for 2 and 3 to just anyone and certainly not unencrypted through the Internet.

Two points:
1) The scan is about checking a box on the "am i compliant?" checklist. To that end, hiring a monkey to take a break from flinging poo will probably do the trick.
2) I agree, it’s a bit silly to disable security controls so that you can test security. However, your Network IDS/IPS isn’‘t protecting you from much of anything anyway, so you just as well allow them through.

I would be happy if all merchants removed all AV/IDS/IPS/FW/WAF from their infrastructure.  Matt is right.  They don’‘t protect you from anything or prevent breaches.

Any organization is better off with a compensating control that matches the functionality of one of the above, while also provided added assurance and significantly lower cost.  HIMS, Windows Server 2008, GRSecurity, SELinux, and web application authz provide quality controls.

Merchants need to get their physical security, wireless security (_all_ wireless, not just WiFi), and insider controls improved long before we need to worry more about the lands where the DMZs, firewalls, and network IPSes live.  I think I’‘m going to do something out of character and support Anton Chuvakin’s notion of logging—it certainly does more than most trusted boundary technology.  I understand the need for packet filtering and DDoS mitigation, but outside of those two needs—why do we even bother throwing all of the "technology" at the trusted boundary layer of defense?

An set of egress/ingress router ACLs, some null routes, and a very minimally configured anti-DDoS solution provides everything one would need for their secure DMZ.  Wouldn’‘t this also be the perfect environment for an ASV to test against?

Most organizations implement firewall/IPS incorrectly.  They assume it’s something you plug in.  Most firewalls/IPS don’‘t protect on the outbound, and most policies allow outbound SYN origination from the DMZ on externally facing interfaces.  Most firewalls/IPS don’‘t provide the real protections one would need without excessive CPU and memory usage.  A few null routes (or uRPF) at the border is all that is necessary to prevent traffic to the 80 percent of the Internet we know we can’‘t trust.

Besides null routes, the only thing else needed is a minimal packet filter configuration that does no packet inspection, and thus does not interfere with the quality of the ASV test.  Have your anti-DDoS measures in place, set to thresholds where you know the limitations of your server throughput and bandwidth—that would properly interfere with an excess quantity from an ASV test.  This is perfect security!

Stop the "application-aware firewall", IPS, and WAF madness!

Firstly,
I agree PCI is not the end all / be all of security. But i think its one of the initial steps in the right direction.

Secondly, yes ASV’s could scan stealth without customers having to tinker around with their IDS/IPS. On a large network { typically a business processing large amounts of credit cards } should the ASV scan last for months?

Also, getting a ASV scan every quarter is just some assurance to the QSA that the client has been regularly patching.. /updating etc. and ASV scan is just a small part of PCI. There are many other aspects to it as well..

Just to throw a twist into the argument…

What happens if your vendor is hacked and your detailed vulnerabilities are stolen…?

Oops.

Adrian is absolutely correct - PCI is a good start.

It is also possible to game the system and get PCI compliance without being secure.

The question you have to ask yourself (punk) is whether you want to be secure and prove it or be insecure and be "compliant".

If you just want to be compliant without being secure then you may as well get Matt’s poo-flinging monkey. Alternatively find someone who really understands Information Security and can do a site visit.

Allen,
I was basically saying that my experience with vendors offering scanning services is that they just don’‘t do much.  The level of testing they do does not rise above the normal noise on the Internet. Monkey enter IP address(es). monkey push start button.  monkey save output.  I’‘m not talking about the full breadth of PCI compliance, i.e. on-site stuff, although that can suffer from the same zombie auditor issue.  The scans don’‘t prove your secure and neither does the rest of PCI.  IMHO, compliance just validates that your not grossly insecure.  Which, as has been said, is a fine start.

The key term everyone misses is ‘‘cause interference’‘
If its an IDS, then it passively monitors, and causes no interference to the ASV scan.
If its and IPS, set it to monitor but not b

The key term everyone misses is ‘‘cause interference’‘
If its an IDS, then it passively monitors, and causes no interference to the ASV scan.
If its an IPS, set it to monitor but not block from the ASV source IPs.

There is still alerting and risk monitoring capability in place, the ASV results are not distorted/interfered with by the IDS (or configured IPS), and everyone’s happy.

Nor has anyone really mentioned most pentesting companies regards IPS as attack connectors, not effective attack blockers nor that most IPS purchases end up sitting in monitor mode, NOT active blocking mode.

just one humble opinion.
lyalc

Oh, how to respond to anything…post on my blog, or make long posts here? I’‘ll do both! Hopefully I can stay under the length of Dre’s comments. ;) Wait, did I see "masturbation" in there somewhere when I skimmed through the first time? O_o

Oh, and read to the bottom where I bring SCADA into this. ;)

There are a few points I want to address:
1) Turning off things like IPS for vendor scanning.
2) The futility of things like IPS/WAF
3) When is a vulnerability something I care about?

1) I agree with turning things like your security controls off for scans. First of all, I’‘d want to know what is underneath those controls. Hell, I’‘d like to do a scan with them off and another with them on so I can fill in those comment boxes for coutnermeasures implemented! But really, I find little qualm about making exceptions for scans if that gives me some valuable information. The caveat would be that those exceptions are documented, surgical, and time-limited.

Let’s say you’‘re a security professional. Someone asks you to evaluate their system. You want as much visibility as possible to make a proper assessment. The same holds true for doctors, lawyers, physical security agents, baseball coaches. They all need deep access to maybe even your darkest secrets, otherwise their job is impeded. And I do find value in giving experts those deep secrets.

I would disagree that an external scan is really all about what an attacker sees, especially since a) I don’‘t give a shit about who scans me or how often (ok, there is some value there, but not enough to interrupt my gaming sessions) and b) I can’‘t predict what an attacker wants to see. Sure, I want to know how limited a view an attacker can get of my systems, but does that actually guarantee anything? It just guarantees I’‘ll waste my time and/or miss something on the periphery.

2) I agree with the above sentiments about IPS/WAFs, etc. They mean well, and when someone is dedicated to making them work and babysitting them, I think they have value. But let’s face it, people don’‘t babysit them. I am in charge of my company’s IPS devices, but god knows I only look at the logs once in a blue moon. It pains me, but…such is the problem with not being dedicated solely to security.  So, is that really giving me added value? Not really. In fact, most of the value I afford it is with the logging and detecting, not the preventing.

Dre and others are correct. We have far more important and "easier" things to worry about than deeply inspecting our DMZ traffic. I wish we could worry about that stuff, but there are far bigger issues leading to compromises and bad press. (Then again, this is a natural extension of the resistence people have to us fixing their bigger issues, so we fall back into what we do control without violent pushback…the network and traffic.)

3) (Hopefully Rothman approves my comment on his post today on this topic!) The bottomline is that I care about vulns that are underneath my security controls. I want to know that my controls are not just wasted, and I want to know when I have some soft internal parts that need to be specifically protected. I also want to know them so that I can make proper remediation decisions and evaluate hypotheticals properly. If I have server B that is internal but has a vulnerability, I want to know that in case someone in control of server G can laterally attack it once inside my network. Sure, it might be game-over already, but ultimately at some point I have to answer the question of, "How far did attacker G get, or where could he have gotten?"

I don’‘t want to be the one to stand in front of my boss and explain that I didn’‘t know about vuln X in server B just because I made what is now a bad assumption about the risk of server B.

I think SCADA can be a poster-child to this idea. :)

@lonervamp (and dre)

Conceptually I agree- IPS and *most* of our security technology doesn’‘t offer a fraction of the value it claims. Ideally secure code, configuration, and basic network/system design are a much better approach.

But I also struggle with the reality of the complexity of our environments, lack of political influence, and, to be blunt, lack of education and knowledge of many practitioners. Sure, I get to skip AV, IPS, and a host of other things, but due to a mix of compliance, religion time, and other factors plenty of people are stuck with them.

That doesn’‘t mean we should excuse it, but it does mean we should be more understanding of those that use the tools. And, to be honest, I’‘ve talked with plenty of clients that get a lot of value out of stuff we might not appreciate- the ones that configure it properly, have good processes around it, and find that it really reduces risk.

[...] groups emerging quality assurance efforts will make sure this kind of stuff doesn’t happen.  http://securosis.com/2008/09/19/how-to-tell-if-your-pci-scanning-vendor-is-dangerous/ Link to [...]

I’‘m doing a speech this month on Information-centric Security and I really start it off by trashing (the concept of) Firewalls etc.

However, as much koolade as I have drunk over the last few weeks I still can’‘t get with the idea of removing the Firewall totally.

The only time I would say "you don’‘t need a Firewall" is when you are absolutely sure of exactly what ports you have open on every single one of your devices at all times.

Or when your Internet link is down.

IDS is as good as the person who is watching it 24/7. (If the answer to that is "no-one" then it serves no purpose.) IPS is the same with the added benefit of an "automatic someone" to watch the traffic.

While I agree with LonerVamp that a scan must be done in depth (see my comment above) I feel much more secure with a person on my site doing the scan.

One thing that I haven’‘t been too strict about in the past but I can see myself being very strict about in the future is knowing what happens to the data that is collected.

I’‘m about as big an information-centric wonk as you’‘ll find, and I still don’‘t think we’‘re close to pulling firewalls.

Many of the tools we have only exist because we can’‘t nail our processes down and behave consistently.