Sophisticated malicious programs used to spy on individuals and businesses represent a “challenge” to security firms, say experts