Deploy, But Verify: Analysing LLM Generated Code Safety

Rasmus Krebs; Somnath Mazumdar

doi:10.1109/PDP66500.2025.00011

Deploy, But Verify: Analysing LLM Generated Code Safety

Rasmus Krebs, Somnath Mazumdar

Institut for Digitalisering

CBS Digital Transformations Platform

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

Abstract

The number of large language models for code generation is rising. However, comprehensive evaluations that focus on reliability and security remain sparse. This study evaluated the Python language code quality generated by five large language models. They are GPT-4-Turbo, DeepSeek-Coder-33B-Instruct, Gemini Pro 1.0, Codex and CodeLLama70 b -Instruct. The evaluation considered three diverse application domains with varying prompt lengths for fair comparison. We found GPT-4-Turbo generated (on average) 4.5% more secure code than a Python code developer with three years of experience.

Originalsprog	Engelsk
Titel	Proceedings - 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2025
Udgivelsessted	Los Alamitos, CA
Forlag	Institute of Electrical and Electronics Engineers Inc.
Publikationsdato	mar. 2025
Sider	13-16
ISBN (Trykt)	9798331524944
ISBN (Elektronisk)	9798331524937
DOI	https://doi.org/10.1109/PDP66500.2025.00011
Status	Udgivet - mar. 2025
Begivenhed	33rd Euromicro International Conference on Parallel, Distributed and Network-based Processing. PDP 2025 - University of Turin, Torino, Italien Varighed: 12 mar. 2025 → 14 mar. 2025 Konferencens nummer: 33 https://pdp2025.org/

Konference

Konference	33rd Euromicro International Conference on Parallel, Distributed and Network-based Processing. PDP 2025
Nummer	33
Lokation	University of Turin
Land/Område	Italien
By	Torino
Periode	12/03/2025 → 14/03/2025
Internetadresse	https://pdp2025.org/

Emneord

Code
LLM
Python
Reliability
Safety
Security

Adgang til dokumentet

10.1109/PDP66500.2025.00011Licens: Ikke-specificeret

Handle.net

Persistent link

Citationsformater

@inproceedings{a2ed80a6d6314607a5924eb7b644a93b,

title = "Deploy, But Verify: Analysing LLM Generated Code Safety",

abstract = "The number of large language models for code generation is rising. However, comprehensive evaluations that focus on reliability and security remain sparse. This study evaluated the Python language code quality generated by five large language models. They are GPT-4-Turbo, DeepSeek-Coder-33B-Instruct, Gemini Pro 1.0, Codex and CodeLLama70 b -Instruct. The evaluation considered three diverse application domains with varying prompt lengths for fair comparison. We found GPT-4-Turbo generated (on average) 4.5% more secure code than a Python code developer with three years of experience.",

keywords = "Code, LLM, Python, Reliability, Safety, Security, Code, LLM, Python, Reliability, Safety, Security",

author = "Rasmus Krebs and Somnath Mazumdar",

year = "2025",

month = mar,

doi = "10.1109/PDP66500.2025.00011",

language = "English",

isbn = "9798331524944",

pages = "13--16",

booktitle = "Proceedings - 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2025",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

note = "33rd Euromicro International Conference on Parallel, Distributed and Network-based Processing. PDP 2025, PDP2025 ; Conference date: 12-03-2025 Through 14-03-2025",

url = "https://pdp2025.org/",

}

Krebs, R & Mazumdar, S 2025, Deploy, But Verify: Analysing LLM Generated Code Safety. i Proceedings - 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2025. Institute of Electrical and Electronics Engineers Inc., Los Alamitos, CA, s. 13-16, 33rd Euromicro International Conference on Parallel, Distributed and Network-based Processing. PDP 2025, Torino, Italien, 12/03/2025. https://doi.org/10.1109/PDP66500.2025.00011

Deploy, But Verify: Analysing LLM Generated Code Safety. / Krebs, Rasmus; Mazumdar, Somnath.
Proceedings - 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2025. Los Alamitos, CA: Institute of Electrical and Electronics Engineers Inc., 2025. s. 13-16.

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

TY - GEN

T1 - Deploy, But Verify

T2 - 33rd Euromicro International Conference on Parallel, Distributed and Network-based Processing. PDP 2025

AU - Krebs, Rasmus

AU - Mazumdar, Somnath

N1 - Conference code: 33

PY - 2025/3

Y1 - 2025/3

N2 - The number of large language models for code generation is rising. However, comprehensive evaluations that focus on reliability and security remain sparse. This study evaluated the Python language code quality generated by five large language models. They are GPT-4-Turbo, DeepSeek-Coder-33B-Instruct, Gemini Pro 1.0, Codex and CodeLLama70 b -Instruct. The evaluation considered three diverse application domains with varying prompt lengths for fair comparison. We found GPT-4-Turbo generated (on average) 4.5% more secure code than a Python code developer with three years of experience.

AB - The number of large language models for code generation is rising. However, comprehensive evaluations that focus on reliability and security remain sparse. This study evaluated the Python language code quality generated by five large language models. They are GPT-4-Turbo, DeepSeek-Coder-33B-Instruct, Gemini Pro 1.0, Codex and CodeLLama70 b -Instruct. The evaluation considered three diverse application domains with varying prompt lengths for fair comparison. We found GPT-4-Turbo generated (on average) 4.5% more secure code than a Python code developer with three years of experience.

KW - Code

KW - LLM

KW - Python

KW - Reliability

KW - Safety

KW - Security

KW - Code

KW - LLM

KW - Python

KW - Reliability

KW - Safety

KW - Security

U2 - 10.1109/PDP66500.2025.00011

DO - 10.1109/PDP66500.2025.00011

M3 - Article in proceedings

SN - 9798331524944

SP - 13

EP - 16

BT - Proceedings - 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2025

PB - Institute of Electrical and Electronics Engineers Inc.

CY - Los Alamitos, CA

Y2 - 12 March 2025 through 14 March 2025

ER -