Blog: Pete Loshin

« Monitoring Telecommunications Traffic Requires BIG Databases | Main | Ready, or Really Ready? »

Data Mining, Terrorism and Open Source Software

If you haven't already seen it, check out this CIO Magazine article, IT Versus Terror right away. And though it isn't highlighted as such, it contains an interesting nugget of a big open source win.

It's mostly about how the Department of Homeland Security is trying to harness data mining for its anti-terror efforts. Unfortunately, the conclusion I drew was that it's easy to waste money on magic bullets like automated terror prevention with little or no oversight.

According to the article, a 2004 GAO survey discovered 199 data mining projects, with 14 explicitly dedicated to catching terrorists and preventing attacks--but the total would surely be higher if you counted the agencies that didn't respond, including the CIA and NSA.

In all that activity, CIO senior writer Ben Worthen was able to uncover only one instance of a data mining success: sorting the sheep from the goats among Guantanamo detainees. The key pieces of software? One was a commercial product from I2 Ltd., for building up relational charts from all data available about the detainees.

The second piece? Open source Proximity, "a system for relational knowledge discovery designed and implemented by the Knowledge Discovery Laboratory in the Department of Computer Science at the University of Massachusetts Amherst."

Share: