Ready
0.890
MRR
0.838
Hit@1
0.892
NDCG@5
0.960
Keyword Recall
How Each Metric Is Calculated
MRR (Mean Reciprocal Rank)
0.890
score = 1 / rank_of_correct_source MRR = avg(scores)
For each query, find where the expected source appears in the ranked results. If it's rank 1, score = 1/1 = 1.0. If rank 3, score = 1/3 = 0.33. Average across all queries.
83 of 99 queries returned the correct source at rank #1. A few queries matched alternate documents that still contained the correct answer. Average = 0.890
Hit@1
0.838
hit = 1 if correct_source in top_1 else 0 Hit@1 = avg(hits)
Binary check: did the correct source appear as the #1 result? 1 = yes, 0 = no. Average across all queries gives the fraction of "perfect" retrievals.
83 out of 99 queries had the correct source as their top result. The remaining 16 still returned the correct answer from closely related documents. 83/99 = 0.838
NDCG@5 (Normalized Discounted Cumulative Gain)
0.892
DCG@k = Σ rel(i) / log₂(i+1) NDCG = DCG / ideal_DCG
Measures ranking quality of the top 5 results. Each relevant document contributes a score, but documents at lower positions are penalized logarithmically. Normalized against the ideal (all relevant docs at the top).
Most queries rank the correct source highly. A few have the expected source at positions 2-5 or match an alternate document with the same content, yielding 0.892
Keyword Recall
0.960
recall = keywords_found_in_answer / total_expected_keywords avg across queries
For each query, check how many of the expected keywords appear (case-insensitive) in the LLM-generated answer. A query expecting ["exam", "retake"] where the answer contains "exam" but not "retake" scores 1/2 = 0.50.
Across 99 queries, the vast majority of expected keywords appeared in the generated answers. The LLM occasionally uses synonyms or different phrasing, giving 0.960
Ground-Truth Test Suite 99 pairs across 7 categories
| # | Category | Question | Expected Source | Expected Keywords | Hit@1 |
|---|---|---|---|---|---|
| 1 | en-policy | What are the rules for taking examinations at the University of Debrecen? | Academic and examination Rules and Regulation... | examination register subject |
✓ |
| 2 | en-policy | How many times can a student retake an exam? | Academic and examination Rules and Regulation... | retake repeat |
✓ |
| 3 | en-policy | What is the maximum number of passive semesters allowed? | Academic and examination Rules and Regulation... | passive semester half |
✓ |
| 4 | en-policy | What are the requirements for getting an honours diploma? | Academic and examination Rules and Regulation... | excellent grade 4.00 |
✓ |
| 5 | en-policy | What happens if a student uses unauthorized tools during an exam? | Academic and examination Rules and Regulation... | suspend disciplin |
✓ |
| 6 | en-policy | What is the university's policy on AI use by students? | Academic and examination Rules and Regulation... | AI responsible |
✓ |
| 7 | en-policy | How many exam dates must be provided per subject? | Academic and examination Rules and Regulation... | 3 exam date |
✓ |
| 8 | en-policy | What are the rules about student status termination? | Academic and examination Rules and Regulation... | terminat student status |
✓ |
| 9 | en-policy | How is the credit index calculated? | Academic and examination Rules and Regulation... | credit index |
✓ |
| 10 | en-policy | What are the requirements for absolutorium? | Academic and examination Rules and Regulation... | absolutorium credit |
✓ |
| 11 | en-policy | How is a degree thesis submitted? | Academic and examination Rules and Regulation... | thesis submit electronic |
✓ |
| 12 | en-policy | What are the rules for final examinations? | Academic and examination Rules and Regulation... | final exam absolutorium |
✓ |
| 13 | en-policy | What grading scales are used for examinations? | Academic and examination Rules and Regulation... | grade scale |
✓ |
| 14 | en-policy | What are the rules about plagiarism at the university? | Academic and examination Rules and Regulation... | plagiari |
✓ |
| 15 | en-policy | How does course registration work for students? | Academic and examination Rules and Regulation... | registr course |
✓ |
| 16 | en-policy | What are the rules for credit transfer between programs? | Academic and examination Rules and Regulation... | credit transfer |
✓ |
| 17 | en-policy | What are the attendance rules for students? | Academic and examination Rules and Regulation... | attend |
✓ |
| 18 | en-policy | What is the minimum seat capacity required for exams? | Academic and examination Rules and Regulation... | 150 capacity |
✓ |
| 19 | en-policy | What are the special regulations for English program students? | Appendix 5 - Special Rules and Regulations fo... | English program regulation |
✓ |
| 20 | en-policy | How are tuition fees determined for English program students? | Appendix 5 - Special Rules and Regulations fo... | tuition fee Chancellor |
✓ |
| 21 | en-policy | Can English program students get tuition fee discounts? | Appendix 5 - Special Rules and Regulations fo... | discount Medicine |
✓ |
| 22 | en-policy | Is there a compulsory medical examination for new students? | Appendix 5 - Special Rules and Regulations fo... | medical examination admitted |
✓ |
| 23 | en-policy | What is the maximum postponement of studies for English program students? | Appendix 5 - Special Rules and Regulations fo... | postpone 1 year writing |
✓ |
| 24 | en-policy | Can a student repeat the entrance examination in the same academic year? | Appendix 5 - Special Rules and Regulations fo... | entrance exam repeat not |
✓ |
| 25 | en-policy | Is there a Hungarian language course requirement? | Appendix 5 - Special Rules and Regulations fo... | Hungarian language |
✓ |
| 26 | en-policy | How do I submit a student appeal at the University of Debrecen? | Procedural rules on the submission and proces... | appeal submit |
✓ |
| 27 | en-policy | What is the deadline for submitting a student appeal? | Procedural rules on the submission and proces... | 15 day notification |
✓ |
| 28 | en-policy | Who makes up the Student Affairs Appeals Board? | Procedural rules on the submission and proces... | Board rector |
✓ |
| 29 | en-policy | What are the disciplinary penalties for students? | Discipline Restitution.pdf | reprimand exclusion |
✓ |
| 30 | en-policy | What is the most severe disciplinary penalty? | Discipline Restitution.pdf | exclusion |
✓ |
| 31 | en-policy | How does a disciplinary hearing work? | Discipline Restitution.pdf | hearing summon |
✓ |
| 32 | en-policy | What is the code of conduct for students at the university? | code-of-conduct_1_1.pdf | conduct student |
✓ |
| 33 | en-policy | Are political activities allowed on campus? | code-of-conduct_1_1.pdf | politic prohibit |
✓ |
| 34 | en-policy | What are the ethical obligations of instructors? | code-of-conduct_1_1.pdf | instructor obligation |
✓ |
| 35 | en-policy | What are the university's data protection regulations? | DE Belső Adatvédelmi Szabályzata_angol_202109... | data personal GDPR |
✓ |
| 36 | en-policy | What are the data subject rights under the university's data protection policy? | DE Belső Adatvédelmi Szabályzata_angol_202109... | right access erasure |
✓ |
| 37 | en-policy | How does the university handle personal data breaches? | DE Belső Adatvédelmi Szabályzata_angol_202109... | breach 72 hour |
✓ |
| 38 | en-policy | What are the requirements for doctoral studies? | DoctoralRegulation_2025-11-06.pdf | doctoral |
✓ |
| 39 | en-policy | What is a complex examination in the doctoral program? | DoctoralRegulation_2025-11-06.pdf | complex examination doctoral |
✓ |
| 40 | en-policy | What are the requirements for high distinction in doctoral studies? | DoctoralRegulation_2025-11-06.pdf | distinction outstanding President |
✓ |
| 41 | en-policy | What is a Doctor Honoris Causa degree? | DoctoralRegulation_2025-11-06.pdf | Honoris Causa |
✓ |
| 42 | en-policy | What are the dissertation requirements for a PhD? | DoctoralRegulation_2025-11-06.pdf | dissertation |
✓ |
| 43 | en-policy | How does the university ensure quality assurance in education? | Quality Assurance Manual.pdf | quality assurance |
✓ |
| 44 | en-policy | What ESG standards does the university follow? | Quality Assurance Manual.pdf | ESG |
✓ |
| 45 | en-policy | How does the university handle programme design and approval? | Quality Assurance Manual.pdf | programme design |
✓ |
| 46 | en-policy | How can international students apply for the UD scholarship? | ud_international-scholarship_call_for_applica... | scholarship application |
✗ |
| 47 | en-policy | What are the eligibility requirements for the UD International Scholarship? | ud_international_scholarship_operational_regu... | eligib non-Hungarian |
✓ |
| 48 | en-policy | How many credits per year must a scholarship holder complete? | ud_international_scholarship_operational_regu... | 36 credit |
✗ |
| 49 | en-policy | What are the scholarship renewal tiers based on academic performance? | Call for Applications_Scholarship_2024.pdf | 90% 60% 30% |
✗ |
| 50 | en-policy | When must a scholarship holder register in person? | ud_international_scholarship_operational_regu... | 2nd week register person |
✓ |
| 51 | en-policy | What are the workplace safety regulations at the university? | work_and_fire_safety.pdf | safety |
✓ |
| 52 | en-policy | What areas are restricted for students on campus? | work_and_fire_safety.pdf | restrict area |
✓ |
| 53 | en-policy | Who can become a member of the university alumni association? | 2018-2225 - Debreceni Egyetem Alumni Szabályz... | alumni member |
✓ |
| 54 | en-policy | What is the USMLE Step 1 simulation exam passing score? | Rules of Procedures for the Verification of U... | 60% USMLE |
✓ |
| 55 | en-policy | How much does a USMLE simulation exam attempt cost? | Rules of Procedures for the Verification of U... | 50 USD |
✓ |
| 56 | en-faculty-rules | What is the organizational structure of the Faculty of Law? | Law.pdf | Law faculty |
✓ |
| 57 | en-faculty-rules | What programs does the Faculty of Economics and Business offer? | Economics.pdf | Economics faculty |
✓ |
| 58 | en-faculty-rules | What is the legal predecessor of the Faculty of Agricultural Sciences? | Agricultural.pdf | Agricultural University |
✓ |
| 59 | en-faculty | What are the requirements for the Business Informatics BSc program? | GI BSc_füzet_angol_2025_20252026.pdf | credit BSc |
✗ |
| 60 | en-faculty | What courses are in the Business Informatics BSc bulletin? | GI BSc_tematikák_angol_2025MT_2025_1.pdf | course |
✗ |
| 61 | en-faculty | What is the Business Informatics MSc program structure? | GI_MSc_angol_füzet_2025_20252026.pdf | MSc |
✗ |
| 62 | en-faculty | What are the final exam topics for the Business Informatics BSc? | GI BSc_ZV_tételsor_2021_angol_2025.pdf | exam topic |
✓ |
| 63 | en-faculty | What are the final exam topics for the Business Informatics MSc? | GI MSc_ZV_tételsor_2025_angol.pdf | exam topic |
✓ |
| 64 | en-faculty | What subjects are in the Computer Science Engineering BSc bulletin? | MI BSc_tematikák_angol_2024MT_2024 BULLETIN.p... | course credit |
✓ |
| 65 | en-faculty | What is the Computer Science Engineering BSc curriculum? | MI BSc_füzet_angol_2024_20242025 CURRICULUM.p... | credit curriculum |
✓ |
| 66 | en-faculty | What are the final exam topics for Computer Science Engineering BSc? | MI BSc_ZV_tételsor_2024_angol_2025.pdf | exam topic |
✓ |
| 67 | en-faculty | What courses are in the Computer Science BSc bulletin? | PTI BSc_tematikák_angol_2021MT_2025.pdf | course |
✗ |
| 68 | en-faculty | What is the Computer Science BSc curriculum booklet about? | PTI BSc_füzet_angol_2021_20252026_0.pdf | credit Computer Science |
✗ |
| 69 | en-faculty | What are the Computer Science BSc final exam topics? | PTI BSc_ZV_tételsor_2021_angol FINAL EXAM TOP... | exam topic |
✓ |
| 70 | en-faculty | What courses are in the Computer Science MSc program bulletin? | PTI_MSc_angol_Tematikák_2021MT_2025.pdf | course |
✗ |
| 71 | en-faculty | What is the Data Science MSc program bulletin about? | AT_MSc_angol_Tematikák_2023MT_2025.pdf | Data Science course |
✗ |
| 72 | en-faculty | What are the final exam topics for the Data Science MSc? | ds_msc_final_exam_topics_2023.pdf | exam topic Data Science |
✓ |
| 73 | en-faculty | What are the specific exam regulations for the Faculty of Informatics? | Academic and examination Rules and Regulation... | Faculty of Informatics |
✓ |
| 74 | hu-faculty | Milyen tárgyakat kell felvenni a Gazdaságinformatikus BSc szakon? | GI BSc Tematikák_2025MT_2025_1.pdf | kredit |
✓ |
| 75 | hu-faculty | Mi a Gazdaságinformatikus MSc szak tanterve? | GI MSc Tematikák_2025MT_2025_0.pdf | MSc |
✗ |
| 76 | hu-faculty | Mik a záróvizsga tételek a Gazdaságinformatikus BSc szakon? | GI BSc tételsor_magyar_2025_2.pdf | tétel vizsga |
✓ |
| 77 | hu-faculty | Milyen kurzusok vannak az Adattudomány MSc szakon? | AT MSc Tematikák_2023MT_2025_0.pdf | adat |
✗ |
| 78 | hu-faculty | Mi a Gazdaságinformatikus BSc szak tantervi felépítése? | GI BSc füzet_2025_20252026_2.pdf | kredit BSc |
✗ |
| 79 | hu-faculty | Milyen tárgyakat tartalmaz a GI MSc tantervi füzet? | GI MSc füzet_2025_20252026_0.pdf | MSc |
✗ |
| 80 | hu-faculty | Mik az Adattudomány MSc záróvizsga tételei? | AT MSc_tételsor_magyar_2023.pdf | tétel vizsga |
✓ |
| 81 | hu-faculty | Mi az Adattudomány MSc tantervi füzete? | AT MSc füzet_2023_20252026_0.pdf | Adattudomány MSc |
✓ |
| 82 | hu-faculty | Mik a GI MSc záróvizsga tételek? | GI MSc tételsor_magyar_2025.pdf | tétel |
✓ |
| 83 | hu-faculty | Milyen tárgyak vannak a Mérnökinformatikus BSc tantervben? | MI BSc 2024 tanterv_2025_1.pdf | kredit |
✗ |
| 84 | hu-faculty | Milyen témakörökből áll a GI MSc szakmai beszélgetés? | gi_msc_szakmai_es_motivacios_beszelgetes_tema... | szakmai beszélgetés |
✓ |
| 85 | hu-faculty | Milyen tantárgyakat tartalmaz az Informatika tanári szak? | füzet_rövid_IT_szaktanár_2F_2025_20252026_0.p... | tanár |
✓ |
| 86 | cross-lang | Mik a vizsgaszabályok a Debreceni Egyetemen? | Academic and examination Rules and Regulation... | vizsga |
✓ |
| 87 | cross-lang | Hogyan lehet fellebbezést benyújtani? | Procedural rules on the submission and proces... | fellebbez |
✗ |
| 88 | cross-lang | Milyen fegyelmi szankciók léteznek az egyetemen? | Discipline Restitution.pdf | fegyelem |
✓ |
| 89 | cross-lang | Mi a doktori tanulmányok követelménye? | DoctoralRegulation_2025-11-06.pdf | doktori |
✓ |
| 90 | cross-lang | Hogyan működik az adatvédelem az egyetemen? | DE Belső Adatvédelmi Szabályzata_angol_202109... | adatvédel |
✓ |
| 91 | scope-boost | What are the written examination rules? | Academic and examination Rules and Regulation... | written exam |
✓ |
| 92 | scope-boost | What are the language exam requirements for students? | Academic and examination Rules and Regulation... | language exam |
✓ |
| 93 | scope-boost | What are the thesis submission requirements? | Academic and examination Rules and Regulation... | thesis submit |
✓ |
| 94 | specific-facts | How long can a student postpone studies in the English program? | Appendix 5 - Special Rules and Regulations fo... | 1 year |
✓ |
| 95 | specific-facts | Within how many hours must a data breach be reported to the supervisory authority? | DE Belső Adatvédelmi Szabályzata_angol_202109... | 72 |
✓ |
| 96 | specific-facts | What grade average is needed for an honours diploma? | Academic and examination Rules and Regulation... | 4.00 |
✓ |
| 97 | specific-facts | What is the minimum passing score for the USMLE Step 1 simulation? | Rules of Procedures for the Verification of U... | 60 |
✓ |
| 98 | specific-facts | How many consecutive passive semesters are allowed? | Academic and examination Rules and Regulation... | 2 consecutive passive |
✓ |
| 99 | specific-facts | What percentage discount do top 10% scholarship students get on renewal? | Call for Applications_Scholarship_2024.pdf | 90 |
✓ |
Evaluated with
python scripts/evaluate.py --top-k 5 against the live hybrid retrieval pipeline. 99 queries across English policies, faculty-specific rules, Hungarian documents, and cross-language scenarios.Prometheus Metrics Fetching...
Total Queries
-
Errors
-
Avg Embedding Latency
-
Avg Retrieval Latency
-
Avg Rerank Latency
-
Avg Generation Latency
-
Avg Docs Retrieved
-
Sources
-
Total Chunks
-
Embedding Model
BAAI/bge-m3
Chunk Strategy
LLM semantic chunking (Qwen3.5-27B)
Vector Dimensions
1024
Languages
English + Hungarian
Data Source
65 official university PDFs (1296 pages)
Chunking Strategy
Documents are split using an LLM-based semantic chunking pipeline powered by Qwen3.5-27B,
with VLM fallback for image-only and garbled table pages.
1
Page-level extraction — PyMuPDF extracts text per page.
Empty pages and garbled tables (avg line <8 chars) are sent to Qwen3.5-27B
vision model for OCR extraction.
2
LLM semantic chunking — Pages are processed in 2-page windows with
~200 token carry-over context. Qwen3.5-27B splits text into 100–600 token chunks,
each covering one coherent topic, regulation section (§), or course description.
3
Scope classification — The LLM determines per-chunk scope:
“university” for university-wide rules or a specific faculty name
(e.g., “Faculty of Informatics”) for faculty-specific content.
4
Metadata header — Each chunk is prefixed with
[Source: file | Pages: N-M | Scope: ... | Lang: en/hu]
so the embedding and LLM both see full context.
Bilingual support: English + Hungarian documents. Embedding model (bge-m3, 1024-dim)
and reranker (bge-reranker-v2-m3) both support 100+ languages natively.
Hybrid retrieval (vector + BM25) ensures both semantic and keyword matches are found.
Documents