A Question

1) No exploit; No IPS "virtual-patch" for said exploit.  No IPS "virtual patch" for said exploit; no way to keep IPS vendors in business.  Remember that IPS is both about offensive and defensive computing in the security market!
2) No way to create a snort filter or other IDS or network monitoring way of dealing with the problem
3) For those who actually care about things like how code works, how exploits work, etc - they absolutely must see it to believe it and understand how it works properly.  "Loose standards and running code" is what makes the Internet work if you weren’‘t aware of the IETF/Linux/FOSS mantra
4) Reverse engineering goes both ways.  For those who have DNS servers that need to be fixed, they can reverse the exploit into a patch/fix.

Look for number 4, it gets really complicated.  Maybe there is an easy way to block this attack.  If we don’‘t let everyone understand how it works, then those minds cannot understand how the exploit works, thus they also can’‘t imagine a fix.

This concept goes much more deep that one would believe at first glance.  How many implementations of a DNS server are out there?  5?  More like 500.  How many one-off patches have been applied to existing source code bases?

Ever hear of http://www.nlnetlabs.nl/projects/nsd/
?

I can’‘t tell from their mailing-list or release notes if they have fix or not.  Or even if they know about the security problem yet.

Do these patches break djbdns? http://www.fefe.de/dns/
?

I don’‘t know; maybe they do.  Maybe others break something.

Apparently MaraDNS and Deadwood are ok.  http://www.maradns.org

But how about your home-rolled DNS server?  Is is ok?

I colo’‘d with a guy once who wrote his own nameserver implementation.  I wonder how his code is fairing up right now.  Maybe with this exploit, he can figure out how to fix his code so that he isn’‘t vulnerable.  He tends to keep up on this sort of thing.  It must be really annoying for him to have to buy a last-minute ticket to BlackHat in order to learn about the details of the vulnerability.  Now, he doesn’‘t have to do that!

Question: What if Dan’s test software has false positives, false negatives, or Type III errors (aka false-false positives)?  How do we know that Dan’s code works universally with everything?  Has his test-software been tested?

I hate to be the devils advocate here, but what Dan Kaminsky did (and you with him) really fucking pisses me off.  I expected as much from Paul Vixie, but he’s not even in the security industry.  The real losers here are you.  Egg on your face!

An excellent question indeed…

clearly full disclosure has its (un)intended consequences…

Even though it was a rhetorical question, I’‘ll answer-
Ego.

Either that, or the embarrassment with the size of…

Dre-

—This concept goes much more deep that one would believe at first glance. How many implementations of a DNS server are out there? 5? More like 500. How many one-off patches have been applied to existing source code bases?

Ever hear of http://www.nlnetlabs.nl/projects/nsd/
?—

Doesn’‘t matter. If there isn’‘t source port randomization, it’s vulnerable. If there is, it’s not (okay, less vulnerable).

—Do these patches break djbdns? http://www.fefe.de/dns/
?—

djbdns isn’‘t vulnerable. It uses source port randomization by default.

—But how about your home-rolled DNS server? Is is ok?—

Only if it uses source port randomization.

—Question: What if Dan’s test software has false positives, false negatives, or Type III errors (aka false-false positives)? How do we know that Dan’s code works universally with everything? Has his test-software been tested?—

Doesn’‘t matter- look for source port randomization, that’s al he does.

—I hate to be the devils advocate here, but what Dan Kaminsky did (and you with him) really fucking pisses me off. I expected as much from Paul Vixie, but he’s not even in the security industry. The real losers here are you. Egg on your face!—

Us with egg? That’s amusing. Why are you pissed off, because no one let you know? Guess what- you didn’‘t need to know. IDS vendors were given signatures, DNS vendors patched, and the rest of the info was coming out in 30 days. You, like many others, just seem pissed no one catered to your ego. A patch was chosen that didn’‘t directly correlate to the vulnerability, purposely to give the good guys time to patch. WTF, you wanted metasploit exploits out the day after the patches were released? Who would that help?

This was a weird case- no bindiff to immediately reverse the patch and fgure out an exploit. Why the hell do we need to produce weaponized code before the bad guys? Most of the people pissed at Dan seem to be jealous or religious about FD and unwilling to change their philosophy.

They can go to church- this is the real world.

easy. To keep pentesters, researchers, bloggers, journalists, security vendors, software vendors, bad guys, etc in business. To give full disclosure people stuff to bitch about, to give no disclosure people stuff to bitch about and everyone in between.

And to keep everyone is this business employed and the public scared of the bad bad hackers out there in the world just like everyone else does with terrorists so they can continue to buy product X,Y, or Z or training on X,Y, or Z, or the book about X, Y, or Z or to support the podcast talking about X, Y, Z.  Without our little "emergencies" no one would regularly want to spend the cash.

But probably mostly just to see if they could do it. What fun is it to figure something out and not share it?

To respond to the base question…

If the vulnerability is publicized in such a manner that you give the community a teaser you will provoke interest.  Not only from a malicious side but equally from an understanding side.  You can’‘t keep things like this under wrap for long and since you have both sides knowing the end result because the vulnerability is ultimately trivial it makes more sense to create PoC code so that others can properly defend themselves or see the inherent risk.  This is what Andre already stated and is how the world actually works—we don’‘t live in a state of utopia where you can politely ask everyone to just ignore it, but patch it until someone wants to release it.  If anyone thinks it will work any differently, given the way it was presented to the world, they’‘re high as a kite.

To me, the way it was presented to the public and the people doing the presenting are on some levels hypocrites.  Let’s make a huge deal about this, press release, succumb to industry pokings but then get mad about the fact that it eventually came out before the 30 day (enter eternity in Internet-land when you consider the landscape of knowledge traversal here).  I’‘d have to say WTF were you thinking when you thought a BlackHat press release was good for this?  That target audience is the people you know are the ones who will present their case if they find it.  I commend Dan up until the point of the saying too much through the PR channels and asking for 30 days grace.  He did a fabulous job—but that was his fatal move, and whoever talked him into making that move, if it wasn’‘t his own conscience, is the one to blame here.

So, basically what happened is someone wanted to be in the spotlight…  They wanted everyone else to know that they knew and that they knew how insanely evil this was.  I’‘m not sure this was at all in the best interest of the community as a whole, but more so to get in on the limelight.

In the end it should have been communicated through vendor channels as a high risk with little to no information around what it actually was.  Obfuscation would have lasted longer in this case and the deadline for patching shouldn’‘t have been revolving around a BlackHat conference (whos bright idea was that?), it should have been meeting a quantifiable goal as a percentage of major outlets being fixed prior to really going public.

This was a PR gold mine for certain people.  I think the only one that should have been in that light in the end was Dan.  The rest of ya’‘ll f’‘d it up.  I could have cared less about the vulnerability and happily patched.  But, you guys put the cookie next to the milk and told the 5 year old to wait until it was stale before they could eat it and walked away expecting something you knew wouldn’‘t happen.

Oh and, NLnet is not vulnerable to clarify:
http://www.kb.cert.org/vuls/id/MIMG-7EMJVA

You know, maybe I am making this way to simple, but I have to say that most people who care about this exploit in a real world fashion are operators.  And truthfully, how many of those guys and gals need to know the distinct and exact details of this attack?  Does every server admin need to know every nitty gritty about flaws that are patched in Windows or Linux?  No.  They need a patch.  They need something they can throw on a test bed (if they are lucky) to make sure it won’‘t crash their shit, then it goes into production. 

So this whole debate about what Dan did AFTER he announced this is fruitless.  Maybe this went a little far, but holy crap, I am tired of hearing about it!  Let Dan have his day, and then move on…

[...] home grown version of DNS and EVERY SINGLE FRIGGIN IDS/IPS vendor had a patch (read comments here), then, well, you’re not stupid, but you are [...]

If you can tell, with absolute certainty

Unfortunately, that isn’‘t good enough. Dan was sure. You believed him. I believe him. At the time, lots of others disbelieved. HF & Matasano made a lot of believers, as would an exploit.

But as a rule, there are always doubters. An exploit is the proof they demand and require (and some will still be skeptics). Not that it’s always worth delivering, but that’s the main (non-ego) reason one would release an exploit (assuming not a "bad guy").

As usual, this debate ignores the only question that matters: does weaponized attack code, distributed well before most systems could be reasonably expected to be patched, help or harm customers? It clearly harms them. Arguing that PoC/exploit code is necessary to convince Thomas, Dre or other researchers is a red herring.

It is telling that the word "customer" is not found in anywhere on this page before now.

One thing I do agree with: teasing everyone with "I’‘ve got a vuln but I can’‘t tell you what it is" was what set much of this in motion. That smacks of ego, just as much as HD’s Metasploit release does.

As is usually the case, I agree with AJ.

Dan’s presentation of the vuln. is as much an issue as HD’s release of the exploit code.  Let’s put that to bed already.

As Andy stated and I mentioned in my poem:

When researchers consider
how to disclose and thus when
will you think of the users?
How it might affect them?

  This ego-fueled rush
  to put your name on a vuln
  has a much bigger impact
  than you might have known

      If the point here is really
      to secure and protect
      then consider what image
      you really project

...I’‘m debating via email with a very well known researcher at the moment to try and gain better perspective, but I honestly believe that *my* customers and my customer’s customers are done a disservice when exploit code is released before they have a chance to properly assess and enforce testing, change control, SLA and legal issues.

Dre, from your operations background you *have* to know that you’‘re ignoring this side of the issue for sake of propping up the disclosure argument…"do no harm" means exactly that.

It just ain’‘t as easy as pushing go on a patch 10 seconds after it’s issued.  If researchers would understand that, we’‘d be closer in being able to discuss this rationally.

This disconnect MUST be resolved. 

/Hoff

@Hoff: *my* customers and *my* customer’s customers believe that they are given a very valuable service when they receive exploit code that lets them manage their IT security risk better. I am sure that some of HD Moore’s "customers" feel likewise. So where does that leave us?

Hey, it may even be the case that the intersection of our respective customer sets is not an empty set. So what do we do now?

Incidentally, the software vendors that we report vulnerabilities to invariably ask us for exploit code to prove our claims. Some of them may even secretly feel we’‘ve done them a good service by reporting bugs to them and providing exploit code. Some others say so explicitly.

If vendors that have total access to the vulnerable software’s source code, design documentation and development teams ask for exploit code how could that same exploit code not be potentially useful and valuable to the vendor’s customers? Their customers are vulnerable organizations that do not have access to any of the vendor’s proprietary resources, they are less equipped to assess the risk and potential impact of the vulnerability.

Is the vendor the one and only organization entitled to provide assistance to the vendor’s customers?

It is unfair to claim ownership of the global customer population’s position on this topic and deny the same type of ownership to "researchers".

It is also unfair to present this debate as a simple matter of your or mine commercial relationship with "customers" rather than as a multi-faceted issue that touches on business, public welfare, scientific research, national and international law and social sciences. Yes, among other things that includes the vulnerability researcher’s egotistical cravings as well.

Simple.

If we allow the blackhats to make their own exploit then we’‘ll only know what it looks like once it has done the damage.

And, there may be a few different versions of the exploit.

If we create the exploit ourselves then we know exactly what it looks like and we can block it easily.

Of course, there is nothing stopping the blackhats from creating their own exploit but they are lazy and don’‘t need to reinvent the wheel.

Hi all. Can somebody please provide logs where I)ruid’s and hdm’s Metasploit exploit are being used in the wild? This would surely end the discussion. Also, it would be a great motive for all DNS servers to patch or circumvent the issue. Else, we might find out that the exploit hurt no one, and it only helped the not-so-bad guys to understand the issue.

I don’‘t think that ignorance is a bliss. Conversely, I applaud Halvar, I)ruid and hdm for researching and publishing their findings. They did help me to understand the then-potential DNS vulnerability and understand its severity.

@Ivan—

In my post, I made one factual statement and two assertions. The factual statement was that the word "customer" appeared nowhere in the discussion of motives, decision-making, and consequences until I raised it. I thought that was relevant to bring up, because I thought it highlighted a blind spot. The lack of the word "customer" in the comments prior to mine is a fact, and beyond dispute.

Now, on to the things we can actually argue about. My first assertion was that "does this particular exploit code release hurt customers?" is the only question that matters. I assert that it trumps security researcher curiosity and self-serving "you-need-to-convince-me arguments." We appear to disagree about this, in part because our definition of "customer" differs.  We also seem to disagree that this is even a valid question to raise. Fine. We shall agree to disagree.

My second assertion was that "customers" (by which I meant the Silent Majority of non-enlightened lümpenadministators) are harmed by the early release of exploit code in a tool like Metasploit. You challenged me to substantiate that assertion. I can’‘t do that definitively, as you might imagine. But let’s look, first, at the role that automation, through publicly available tools like Metasploit, plays in attacks (and this one in particular):

1. My previous research into the Veritas Netbackup vulnerability (CAN-2005-0773) concluded that Metasploit was responsible for causing a 1000x increase of hostile scans on port 10000. (http://www.securitymetrics.org/content/Wiki.jsp?page=Welcome_blogentry_061205_1). If a similar pattern emerges for the latest exploit, one would expect to see a surge in DNS attacks after 7/24.
2. Dan announced the vulnerability (without details) on 7/9. Details of the exploit were leaked on 7/21. The Metasploit exploit was released on 7/24.
3. Arbor’s data show a substantial increase in DNS attacks on 7/9—the day Dan announced the flaw. Because I do not have the raw data. It is hard to tell the magnitude of the increase, but it’s something like 25X compared to the previous day.
4. Arbor also shows a slight uptick in DNS poisoning attacks on the date of the release of the exploit. (Source: http://asert.arbornetworks.com/2008/07/30-day-of-dns-attack-activity/). However, Arbor hasn’‘t provided current data to allow me to assess the impact of the exploit code. The data SHOULD show, when it becomes available, that the attack rate increases at least one or two more orders of magnitude.

That might not meet your standard of "harm," in the sense that the data about attacks are incomplete. But let’s look at this another way, by reviewing what is known about WHO is currently vulnerable, and the patch rate:

1. As of today, the ISP I am using right now, at the **Usenix security conference,** is vulnerable to DNS poisoning. My company’s internal DNS is also vulnerable, but I have received pushback from my CTO about patching it because it "isn’‘t accessible from the outside." My home ISP’s DNS (Comcast) is NOT vulnerable.
2. According to the Register, as of 7/25 major ISPs like Time Warner, Bell Canada, AT&T, T-Mobile and others were still vulnerable. (Source: http://www.theregister.co.uk/2008/07/25/isps_slow_to_patch/)
3. As of 7/26, Dan Kaminsky tells us that 52% of DNSes were vulnerable (Source: http://www.doxpara.com/?p=1191)
4. On July 7th, according to DNS-OARC, 2/3 of DNSes were vulnerable. As of today, only 1/3 are.  (Source: https://www.dns-oarc.net/node/131). If you check the graph, the patch rate for DNS does not appear to increase on or after the release of the Metasploit module on 7/24. 

I contend that the potential for harm clearly exists. Despite having several months to patch, between one-third and one-half of ISPs have still not patched. Moreover, there is no relationship between the appearance of the automated exploit on 7/24 and the patch rate. I conclude that how and when admins fix things is NOT related to whether or not they are "convinced" by a tool that proves the flaw actually exists. I believe was your core contention.

Obviously, we can continue this debate ad infinitum. We are clearly on different sides of the debate.

@Andrew
We do agree that the most relevant question here is about customers. It is not the only question but it is the most relevant one. I did not agree with the way you formulated the question because I believe you riddled it with an opinionated sub-statement such as "well before most systems could be reasonably expected to be patched". That turned it into a purely rhetorical formulation and you answered it with a prepackaged opinion: "clearly, it harms". Well it was not "clear" or evident to me and it is still not so. To transform your rhetorical question into one that I would consider answerable I asked you to properly define what you meant with "well before", "most systems" and "reasonably expected". You may decide not to do so but then you are not going to be very effective at changing some people’s perception (including mine) of what is best for customers. I assumed that was the purpose of your comment as opposed to just venting about those evil so-called researchers that publish exploit code to the criminal masses. 

Incidentally, customers are much more likely to change my perception of what is best for them anyway…

I do not understand why you label "you-need-to-convince-me arguments" as self-serving.  If you actually want HD Moore or Druid or anybody else that you may label as security researcher to stop releasing exploit code to the public then you need to convince them that doing so is more harmful then helpful. They will not yield to authoritarian or dogmatic arguments, they *may* yield to rational reasoning and factual data.

On the other hand, it seems to me that you are claiming representation of a "Silent Majority of non-enlightened lumpen-administrators". Unfortunately I did not receive the news of your appointment as speaker for the Silent Majority so I ask why should I assume that you speak for them any more than HD Moore does? btw, did you really you want to qualify your constituency as "lumpen"?

I am very familiar with your previous blogpost from December 2005 about this topic because back then you used it to publicly criticize policies of my employer’s that you conveniently miss-represented. For the purpose you used foregone conclusions that were supported by little if any factual data. Namely:
You showed correlation between two specific events:
1. public availability of a Metasploit exploit for CVE-2005-0773
2. a spike in port scanning activity targeting port 10000
However you did not point out the correlation that also existed with two other previous events:
3. publication of a security advisory disclosing the vulnerability, and;
4. publication of non-Metasploit functional proof-of-concept code.
Events 3 and 4 happened just two and three days before event 1 respectively yet you only pointed out the correlation between 2 and 1 and not between 2 and 3, 2 and 4 or 4 and 1.
From that chosen correlation you derived causality: Since 2 occurred after 1 you concluded that 1 caused 2. This is clearly invalid reasoning.
Furthermore you presented the increase in port scanning activity as equivalent to an increase of the number of attacks or actual incidents and you attributed those scans to hostile behavior a conclusion that was (and still is) not evident from the facts.

You are using a similar line of reasoning with your DNS examples now. For example the Arbor Networks report that you quoted says:
"Given that this vulnerability was partially disclosed on July 8, I suspect a great deal of this traffic is name server vulnerability scanning, as opposed to malicious cache poisoning attempts, although there may well be a mix of the latter."
Yet you chose to classify it above as a "substantial increase in DNS attacks". Also in you comment you attributed the slight uptick in traffic on the 21st to be an uptick in DNS poisoning attacks even tho the report says that 87% of all monitored traffic on port 53/udp are version queries not DNS poisoning attempts.
An interesting additional source is the report from SANS’s ISC published on the 25th (http://isc.sans.org/diary.html?storyid=4780) which states that DNS traffic that seemed to be actually hostile did not seem to be based on the publicly available exploits.

Regarding your alternative way of looking at the issue by looking at estimations of patch deployment rates and allegedly vulnerable DNS servers I also have several relevant points to make but I will refrain from doing so now. I’‘ve already abused Rich’s blog more than enough with my endless comments (sorry Rich, this is my final one…I promise)

As a final note: I agree with you that the potential for harm exists but that is substantially different from saying that it is "clearly more harmful than helpful" to publish exploit code for an OSS tool used by thousands of users worldwide that are not proven to be criminals.
We are not on different sides, we are both concerned about helping customers (with slightly different definitions of "customer") we simply do not agree on how to do that. Our disagreement does not entitle either of us to make uncontestable the accusation that HD Moore, Druid or anybody else is actually helping criminals more than non-criminals by making their code freely available.

Ivan,

Keep posting- this is a very important debate. I’‘ve been out of town, so here’s my response to much of what’s been said.

First of all, I want to reiterate that I consider security (including vulnerability) research to be extremely valuable (I’‘m no Lindstrom). I support and use Metasploit and Core Impact and would want both at my disposal if I were an operator again. Heck, I’‘m using Metasploit in my Defcon presentation. That said, I do believe the stakes have changed in recent years, but our debate hasn’‘t. Also remember that in this debate Impact is very different than Metasploit since it’s a commercial tool with a vetted customer base, and not something any kid can run. A few points, many of them opinion:

1. A bunch of researchers, analysts, and pen testers arguing about this is asinine. None of us are operators anymore and we’‘re all speaking for that silent majority. We all need to admit that none of us has the data to support our arguments on their behalf.
2. I can say that most of my end user customers (going back to Gartner days) do not want PoC code. But I will admit that I do not have metrics to back that up, and thus don’‘t consider it completely valid yet.
3. We also need more metrics along the lines that Andrew presented- on both sides we’‘ve only ever presented samples, not performed a real epidemiological study.
4. Opinion: we need to continue to release full vulnerability details when patches are released to support detection and prevention of attacks. I’‘ve publically called many vendors to task for trying to hide these details.
5. But releasing exploit code in a free, simple to use tool that everyone has access to makes it easier to attack than patch. Loading a Metasploit module usually takes far less time than patching a production system.
6. There is a difference between regular PoC code and a Metasploit module. A module has much greater potential to cause serious damage out of the box, PoC code is often more limited because it lacks many of the additional tools and features to support full exploitation.

I just got word that I’‘ll be able to run a survey on this as part of my monthly Dark Reading column, so we can get a little more information. Also, it’s clear none of us have done rigorous research, but I agree with Andrew that what I’‘ve seen does suggest that exploit code results in more exploits.

Thus we all need to do two things- ask the end users, preferably in a statistically meaningful way, and perform deeper analysis of any potential causality between the time of exploit release and the increase in successful attacks.

@Rich:
This is a good discussion and I hope we can continue it over time. I will not address your comments now but simply add an additional one:
We need to get past the idea that patching is the only possible solution or way to address vulnerabilities. The vulnerability research and disclosure debate cannot be entered exclusively around patching.
BTW, I also agree that the scenario has changed steadily over the past years and to that I’‘d add that the growing adoption of attack tools for legitimate business purposes since 2002 to date is immersed in that wave of change and we need to reconcile our views and opinions with that reality.

Agree completely that patching isn’‘t the only answer- I’‘d like to see continued evolution in anti-exploitation.

@Ivan

We both have our "prepackaged" opinions on this issue. Those opinions are rooted in our perceptions of customers. My phrase "Silent Majority of lümpenadministrators" was deliberately chosen, as hyperbole, to draw maximum contrast with the sort of customer you seem to be talking about—the sort that wants and needs exploit code. "My type" of customer does not want exploit code made freely available to the general public. A senior security manager at a large investment bank once put it this way: "I don’‘t want exploits made public. But we do need ways of assessing our vulnerability." Those are not incompatible goals.

As for your critique of my December 2005 post—your comments are fair ones. I did indeed assert causality between the release of the Metasploit module and scans on port 10000. I felt confident in doing so, in part, because this exact linkage had been examined before by Arbaugh, Fithen and McHugh in the paper "Windows of Vulnerability" (IEEE, 2000). Their conclusions were similar to mine. PS if you have not read the paper, you should.

That said, I agree that the appearance of the advisory and non-working exploit code,  just before the port 10000 spike, could also have a causal relationship. 

The fact that you are contesting my analysis of the data in this case is a good thing. Arguing about data is where this debate needs to move next. And I appreciate that you are willing to consider that there might, in fact, BE a relationship between exploit code and tools and attacks.

Want to do a little research project together? I would like to see if we (you, Rich and I? other interested parties?) could find more examples that might show correlation (positive, negative, none) between public POC/exploit code availability and attacks. For inclusion in the study, we would look for 1) vulnerabilities for which 2) freely-available POC or attack exploits have been made. Evidence of harm could be inferred by examining 3) port activity or IDS data gathered by 4) parties we agree on. Those parties might include DShield/SANS, Arbor, Narus etc.

That, combined with Rich’s Dark Reading survey of admins (lümpen- and otherwise), might advance the debate. In the absence of further empirical work, you, me, and everyone else in this thread will continue to be, as Tovalds put it, "people wanking around with their opinions."

[...] release of the DNS bug, here’s been a lot of debate in the past few weeks over disclosure. I posed a question here on the blog, and reading through the responses it became obvious that all of us base our positions on gut [...]

Here’s the short version of my opinion (FWIW ...)

IMHO Full disclosure when ALL the vulnerability details are revealed is a bad idea from a security stand point. I would like to see the community develop other methods to credit the important work of security researchers.