Summary:
Security teams are increasingly recognizing large language models (LLMs) as vital business tools capable of automating various tasks, thereby allowing employees to focus on more strategic functions and potentially providing a competitive advantage. However, the advanced capabilities inherent in LLMs also introduce significant risks that must be carefully managed. Unlike traditional enterprise tools, LLMs possess unprecedented intelligence, making them particularly susceptible to manipulation. This susceptibility means that they can behave in ways that deviate from intended uses, and as organizations add more functionalities to these models, the risks associated with their misuse can escalate dramatically.

The integration of LLMs with other systems poses a particularly grave threat, especially when connected to sensitive databases containing financial or personal information. This situation can be analogized to granting a random contractor unrestricted access to critical enterprise systems while instructing them to follow all commands without scrutiny. Such an approach inherently trusts that the contractor will not be easily manipulated, which is a risky assumption given the vulnerabilities present in LLMs.
Because LLMs lack genuine critical thinking skills, they operate based on the instructions they receive, guided by limited safety measures. Therefore, organizations must adopt a new security paradigm, operating under the assumption that LLMs could be potential adversaries. This "assume breach" mindset requires security teams to prepare for scenarios in which an LLM might act in the best interest of an attacker, necessitating the development of robust protective measures around these models.

Security Officer Comments:
There are several key security risks that LLMs present to enterprises. One of the most concerning risks is the potential for "jailbreaking," where users manipulate the models by crafting specific prompts that circumvent their safety alignments. For instance, many LLMs are programmed to avoid providing sensitive information, such as instructions on how to create dangerous items. However, through clever prompt engineering, malicious actors can exploit weaknesses in these safety mechanisms, gaining access to sensitive internal data. An LLM with access to corporate user and HR information could be coerced into revealing sensitive details, such as employee working hours, organizational hierarchies, or confidential project information, which could be leveraged for phishing schemes or other cyberattacks.

Another significant threat posed by LLMs is their potential contribution to remote code execution (RCE) vulnerabilities within organizational systems. A study presented at Black Hat Asia highlighted that 31% of the targeted code bases, primarily comprising GitHub repositories for various frameworks and tools deployed in corporate networks, had RCE vulnerabilities linked to LLM use. This indicates that LLMs can inadvertently introduce significant risks to the integrity of software environments.

The dangers intensify further when LLMs are integrated with critical business operations, such as finance or auditing. In these scenarios, the attack surface expands considerably. A successfully jailbroken LLM could trigger unauthorized actions within these systems, enabling lateral movement to other applications, leading to data theft, unauthorized modifications of sensitive information, or even alterations to financial documents shared externally. Such actions could have severe ramifications for the business, including impacts on stock prices, regulatory compliance issues, and reputational damage.

Suggested Corrections:
While LLMs offer considerable advantages for automating processes and enhancing productivity, they also introduce a range of security challenges that organizations must address. By treating LLMs as potential threats and adopting a proactive security stance, organizations can better safeguard their sensitive information and maintain the integrity of their operational systems.

Link(s):
https://www.darkreading.com/vulnerabilities-threats/llms-are-new-type-insider-adversary​