IT Tech blogging

RTX 3080TI 를 사용해서 LLM 모델 llama-bench 로 벤치마크 테스트를 수행했다.

1. kanana-nano-7b
2. exaone-

kanana-nano-7b-Q8

$ llama-bench -m kanana-nano-2.1b-instruct.Q8_0.gguf \
 -ngl 45,47,52,55,57,60 -n 1000

Device 0: NVIDIA GeForce RTX 3080 Ti, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	test	t/s
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	45	pp512	11015.00 ± 48.73
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	45	tg1000	184.19 ± 14.65
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	47	pp512	10965.95 ± 158.97
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	47	tg1000	183.49 ± 15.45
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	52	pp512	10891.63 ± 105.68
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	52	tg1000	183.41 ± 14.09
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	55	pp512	10663.79 ± 150.73
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	55	tg1000	183.14 ± 13.67
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	57	pp512	10784.13 ± 43.64
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	57	tg1000	183.82 ± 12.81
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	60	pp512	10772.47 ± 98.95
llama 8B Q8_0	2.07 GiB	2.09 B	CUDA	60	tg1000	184.45 ± 13.08

ExaOne-deep-7.8B-Q8

ExaOne-deep-7.8B-Q8 버전

(Deepseek_R1) qkboo:/mnt/d/LLM_Local$ llama-bench -m /mnt/e/LLM_Run/EXAONE-Deep-7.8B-Q8_0.gguf  -ngl 27,30,33,35,40 -n 1000

Device 0: NVIDIA GeForce RTX 3080 Ti, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	test	t/s
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	27	pp512	2224.32 ± 55.56
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	27	tg1000	14.72 ± 0.28
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	30	pp512	3048.43 ± 61.25
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	30	tg1000	27.83 ± 0.43
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	33	pp512	4376.37 ± 16.93
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	33	tg1000	83.44 ± 3.36
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	35	pp512	4509.89 ± 19.90
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	35	tg1000	81.71 ± 3.20
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	40	pp512	4414.70 ± 12.80
exaone 8B Q8_0	7.74 GiB	7.82 B	CUDA	40	tg1000	81.99 ± 3.09

ExaOne-deep-2.4B

(Deepseek_R1) qkboo:/mnt/d/LLM_Local$ DLLAMA_CUBLAS=on llama-bench -m /mnt/e/LLM_Run/EXAONE-Deep-2.4B-BF16.gguf -ngl 35,39,43,47,5
0 -n 1000
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080 Ti, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	test	t/s
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	35	pp512	5185.28 ± 22.56
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	35	tg1000	117.92 ± 16.01
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	39	pp512	5273.68 ± 12.42
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	39	tg1000	123.80 ± 13.00
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	43	pp512	5065.21 ± 472.49
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	43	tg1000	125.65 ± 14.67
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	47	pp512	5488.28 ± 16.52
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	47	tg1000	125.00 ± 18.64
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	50	pp512	5481.39 ± 20.48
exaone ?B BF16	4.97 GiB	2.67 B	CUDA	50	tg1000	128.38 ± 14.02

build: 98f6b0fd (4676)

ExaOne-deep-2.4B-Reasoning-MAX-NEO

$ DLLAMA_CUBLAS=on llama-bench -m /mnt/e/LLM_Run/EXAONE-Deep-2.4B-Reasoning-MAX-NEO-D_AU-Q8_0-imat.gguf \
 -ngl 50,57,65,70,78 -n 1000

Device 0: NVIDIA GeForce RTX 3080 Ti, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	test	t/s
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	50	pp512	10672.83 ± 289.15
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	50	tg1000	175.51 ± 3.57
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	57	pp512	10791.99 ± 153.87
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	57	tg1000	163.48 ± 29.00
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	65	pp512	11039.36 ± 118.98
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	65	tg1000	173.37 ± 1.97
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	70	pp512	10539.79 ± 56.69
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	70	tg1000	155.61 ± 23.00
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	78	pp512	10077.84 ± 57.26
exaone ?B Q8_0	3.10 GiB	2.67 B	CUDA	78	tg1000	159.27 ± 21.50