Student uses prompt injection to expose Bing Chat's hidden 'Sydney' system prompt
Liu told the chatbot to 'ignore previous instructions' and asked what preceded them, prompting it to disclose rules telling it to keep its Sydney codename confidential.
Security & misuse