The results for the o1-preview model's performance on the Bigcodebench Hard Instruct evaluation are in. o1-preview scored 27%, slightly higher than GPT-4 Turbo's 26.4%, making it the highest-scoring single model evaluated by Catena Labs. The evaluation was conducted using the provided Docker setup. Other notable scores include GPT-4o at 25.0% and Claude 3.5 Sonnet at 24.3%. However, o1-preview's performance on other benchmarks showed mixed results, with an average score of 28.8% and a significantly lower score of 23.0% on detailed instruction tasks. Additionally, o1-preview achieved 34.5% on BigCodeBench-Hard, but only 10% on an internal codegen benchmark where GPT-4o scored 38%.