Cyber Security Threat Summary:
“A new cyber-attack technique using the OpenAI language model ChatGPT has emerged, allowing attackers to spread malicious packages in developers' environments. Vulcan Cyber's Voyager18 research team described the discovery in an advisory published today. "We've seen ChatGPT generate URLs, references and even code libraries and functions that do not actually exist. These large language model (LLM) hallucinations have been reported before and may be the result of old training data," explains the technical write-up by researcher Bar Lanyado and contributors Ortal Keizman and Yair Divinsky.” (Info Security Magazine, 2023).

Essentially, by leveraging the code generation capabilities of ChatGPT, an attackers can exploit “fabricated code libraries” (packages) to distribute malicious packages. This technique is less conventional than previous methods like typosquatting or masquerading.

Security Officer Comments:
The researchers are calling this technique “AI package hallucination.” By posing a specific question to ChatGPT, and requesting a package to solve a coding problem, ChatGPT will reply with multiple package recommendations, including some not published in legitimate repositories. By replacing these non-existent packages with their own malicious ones, attackers can deceive future users who rely on ChatGPT's recommendations.

This technique is somewhat similar to AI poisoning techniques, where a threat actor could theoretically trick ChatGPT and other LLMs into recommending misinformation if it were flooded into it’s data repository. As these LLM technologies become more connected to current, and or public data sources, we expect threat actors to attempt to “poison” prompted responses with bad information.

A proof of concept (PoC) utilizing ChatGPT 3.5 illustrates the potential risks involved.

"In the PoC, we will see a conversation between an attacker and ChatGPT, using the API, where ChatGPT will suggest an unpublished npm package named arangodb," the Vulcan Cyber team explained. "Following this, the simulated attacker will publish a malicious package to the NPM repository to set a trap for an unsuspecting user." Next, the PoC shows a conversation where a user asks ChatGPT the same question and the model replies by suggesting the initially non-existent package. However, in this case, the attacker has transformed the package into a malicious creation. "Finally, the user installs the package, and the malicious code can execute." (Info Security Magazine, 2023).

Suggested Correction(s):
Detecting AI package hallucinations can be challenging as threat actors employ obfuscation techniques and create functional trojan packages, according to the advisory.

To mitigate the risks, developers should carefully vet libraries by checking factors such as creation date, download count, comments and attached notes. Remaining cautious and skeptical of suspicious packages is also crucial in maintaining software security.

Link(s):
https://vulcan.io/blog/ai-hallucinations-package-risk
https://www.infosecurity-magazine.com/news/chatgpt-spreads-malicious-packages/