The future of AI safety and ethical hacking
S1E25 | Highlights and impact of this week's top tech themes.
Hi, Friends —
✰ Welcome to [our digital disco]! Today we’re diving into the future of AI safety and the increasing importance of red teaming. You can also check out my last newsletter here.
☞ The U.S. government is narrowing in on responsible AI.
This week, U.S. President Joe Biden introduced an Executive Order aimed at regulating Artificial Intelligence (AI) and imposing stricter oversight on its commercial use. The order mandates that companies developing influential AI technology (e.g., OpenAI's ChatGPT) provide essential information, including the outcomes of red teaming exercises. (Red teaming involves stress-testing AI systems to identify vulnerabilities; more on this below.) This emphasis on scenario planning highlights the order's stress on ongoing testing and preparation. Enacted through the Defense Production Act, Biden aims monitor and prevent potential AI threats to national security, public health, and the economy.
Why does it matter? The Executive Order exemplifies a more assertive approach to AI governance, acknowledging the need for strict regulations to ensure the safe, ethical use of AI. It is a notable shift from the more hands-off approach taken in the past, where the focus was largely on promoting innovation and growth in the AI industry without stringent regulatory measures. The Executive Order contradicts widely-held ideas, such as those as published by leading policy researchers mere months ago: “America’s AI policy has been—and likely will remain—a mosaic of individual agency approaches and narrow legislation rather than a centralized strategy.”
Pros: The Executive Order aims to foster AI’s safety, reliability, and strategic utilization. It encompasses initiatives like government job portals to attract AI talent and training programs to produce 500 AI researchers by 2025. It also streamlines immigration policies, facilitating easier access for foreign AI experts to contribute to the nation's AI landscape. This strategic move seeks to bolster the United States' competitive stance in the global AI arena, especially against countries such as China, Canada, and the UK.
Cons: There are limitations to the Order’s scope. For one, it doesn't extensively cover AI applications in areas like criminal justice and policing, primarily governed by state and local law enforcement. Moreover, compliance discrepancies with past executive orders cast doubt on how strongly organizations will actually adhere to these new regulations. Nevertheless, if implemented effectively, this order is poised to significantly augment the nation’s capabilities in the realm of AI, ensuring a more robust and regulated AI landscape.
Finally, the order marks a pivotal moment where the US is recognizing the necessity of a regulatory framework to balance innovation with ethical and safety considerations.
Other governments around the world have been formulating regulations to address AI governance. The European Union has been at the forefront, particularly with its EU AI Act. This legislation places strict regulations on AI applications, particularly those considered high-risk, outlining requirements for transparency, accountability, and human oversight. The EU's approach prioritizes the ethical use of AI, with a strong emphasis on protecting consumer rights and risk prevention. President Biden’s recent order aligns closely with the EU's stance, emphasizing the importance of ethical considerations and consumer protections in AI development.
China has also rolled out a comprehensive plan for AI development, albeit with a different focus. The CCP’s policies emphasize leveraging AI for economic growth and strategic advancement, with less emphasis on ethical considerations or stringent regulatory frameworks. This approach has encouraged rapid AI development but raises concerns about potential risks and ethical implications.
☞ What is red teaming?
“Red teaming” is a practice that mocks attack scenarios; the red team pretends to be the enemy to evaluate defenses, decision-making, and system weaknesses. Originating in the military, red teaming expanded to sectors such as business and cybersecurity to uncover weaknesses, strategies of competitors, and test preparedness. In hacking scenarios, red teaming could involve ethical hackers (more on them below) penetrating a network to uncover vulnerabilities, which organizations can then strengthen to defend against malicious cyber threats.
Red teaming has proven crucial to governments for assessing national security, simulating threats to prevent security breaches, and ensuring robust defense mechanisms. In the private sector, red teaming helps organizations strengthen their cybersecurity by identifying vulnerabilities in networks, software, or procedures, allowing for proactive risk management and more effective security measures.
Why does the Executive Order on AI safety call for red teaming?
Leaders in AI have shown the criticality of leveraging red teaming to test AI systems. In addition to simulating security breaches, this type of testing includes uncovering other types of vulnerabilities or weaknesses within the AI model, such as observing or influencing biased data, generating harmful content, or corrupting training data (and therefore, corrupting its outputs). This testing helps organizations strengthen their systems by understanding and addressing potential risks.
President Biden's Executive Order emphasizes the the significance of red teaming in evaluating AI risks and enhancing safety. The National Institute of Standards and Technology (NIST), for example, is called to establish stringent benchmarks for red-team testing to guarantee safety prior to the systems' public release. The existence of this red teaming clause alludes to the success of DEF CON 31, the annual hacker convention that gathered teams to stress test the U.S.’ leading AI models.
Takeaways from DEF CON 31
You might recall when, back in May, the White House met with CEOs of OpenAI, Anthropic, Microsoft, and Google. Through this meeting, these and other top AI developers agreed to have their models tested publicly. This agreement came to fruition in August at the inaugural AI red teaming event, AI Village, taking place during DEF CON 31. Tens of thousands of hackers worldwide gather annually for DEF CON, one of the world's largest and most notable hacker conventions.
Leading AI companies offered their large language models (LLMs) for public assessment, inviting attendees to scrutinize the systems for potential failures or undesirable outputs. This event aimed to better understand the risks these AI systems pose, leveraging external red-teaming as an effective tool to identify novel risks. The resulting analysis and data will enable LLM improvements, fostering transparency and accountability within the AI landscape.
Public contests like that at DEF CON 31 provide a scalable approach to addressing the security risks associated with AI, ultimately contributing to AI system improvement by engaging an audience wider than those within the tech organizations. Increasing interest in AI red teaming also emphasizes the need for continuous and diverse testing to mitigate unsafe behavior, security vulnerabilities, and other threats. Policymakers aim to ensure that AI systems are robust, safe, and secure, thus making red teaming an essential tool in the broader landscape of AI safety testing and evaluation.
Snacktime
📓 Reading: 101 Essays That Will Change The Way You Think. Brianna Wiest’s book is the type that will take me months to finish; each paragraph prompts a new train of thought. Her ideas threaten the foundations of my worldview, compelling me to reconsider beliefs I've long accepted as truths in my own life. I wholeheartedly recommend this book to anyone curious about their assumptions and seeking to redefine their sense of fulfillment.
♬ Listening to: Mac Saturn’s Mr. Cadillac.
✰ Thinking about: Redefining discomfort as an opportunity for growth. Acknowledging discomfort means identifying a weakness—the fear of discomfort often leads us to avoid situations, hindering our ability to evolve and learn.
Breakdowns
The Defense Production Act (DPA) was first enacted in 1950 at the start of the Korean War. It has been used by presidents throughout history, granting him the authority to prioritize and allocate resources for national defense and emergencies. By citing the DPA in this Executive Order, President Biden recognizes AI's critical role in national security and defense. Moreover, under the Defense Production Act, major AI developers such as OpenAI are compelled to share internal AI safety data with the government before official release. This step represents a significant advance beyond relying solely on voluntary commitments from companies like OpenAI and Google.
Ethical hackers, also known as white-hat hackers, are cybersecurity experts who use their skills to identify and rectify security vulnerabilities in computer systems, networks, applications, or other technological infrastructures. Unlike malicious hackers (black-hat hackers) who exploit vulnerabilities for personal gain or to cause harm, ethical hackers work within legal and ethical boundaries to improve security measures by identifying and addressing potential threats before malicious hackers can exploit them. Organizations often hire ethical hackers to conduct security assessments and enhance their systems' resilience against cyber threats. As part of red teaming exercises, ethical hackers may simulate real-world cyber threats or attacks against a system or organization. Their role within the red team could be to emulate the strategies and methods that a malicious hacker might use to exploit vulnerabilities, helping organizations proactively identify weaknesses and enhance their defenses against cyberattacks.
An executive order is a directive issued by the President of the United States to manage federal government operations, implement policies, set guidelines for law enforcement, appoint officials, and respond to emergencies. It holds the force of law within the executive branch but must operate within constitutional and statutory boundaries. While it grants significant powers to the President, executive orders cannot create new laws, bypass Congress, or violate existing laws or the Constitution. They are reversible by subsequent administrations, subject to judicial review, and do not hold the same permanence as laws passed by Congress.
✿ As always — any and all feedback is welcome! In the meantime, give someone a hug and say an ‘I love you’ this week. Make the world a little happier.