THESEFAWAY Products PDFreak AI
Research prototype · BSc Computer Science, Brunel University London · 2025
PDFreak AI.
Explainable PDF malware detection for small security teams. It tells an analyst whether a PDF is malicious, what kind of threat it is, which MITRE ATT&CK technique it maps to and what to do next, without sending the file to anyone else’s cloud.
- Random Forest
- Mistral 7B
- FAISS retrieval
- MITRE ATT&CK
- Ghidra
- FastAPI
- React
- Docker
- Accuracy
- 83.2%
- Random Forest, held-out test set
- ROC AUC
- 0.928
- How well it separates benign from malicious
- ATT&CK mapping
- 90%
- Of 50 reports cited an appropriate technique
- Classifier time
- < 2 s
- Per file, on CPU, no GPU
01 · The problem
PDFs are an easy way in.
PDFs can carry JavaScript, embedded files, launch actions and obfuscated payloads, and they arrive by email every day. Signature-based antivirus catches known samples but misses new or disguised ones.
Large security teams answer this with commercial sandboxes and cloud AI services. Small SOC teams often can’t: the tools are expensive, and uploading a suspicious file, which may contain sensitive business data, to a third party is a privacy decision in its own right. They are also left with a verdict and no explanation.
PDFreak AI set out to answer three questions for a small team, on its own hardware: is this file malicious, what kind of threat is it, and what should I do about it?
02 · How it works
Classify first, explain second.
A fast, interpretable classifier handles every file. The language model, the expensive part, only runs on the files it flags.
Pipeline12 stages · 2 paths
Upload
A PDF or ZIP goes to a FastAPI endpoint and lands in an isolated working directory inside Docker.
Static analysis
PDFiD, peepdf and pdf-parser pull out structure: objects, streams, scripts, and tags such as
/OpenActionand/JavaScript.Dynamic analysis
In parallel, Ghidra’s headless analyser looks for shellcode patterns, and a sandbox VM watches process creation, file and registry changes, and network activity.
Features
Over 45 raw signals become one vector, then cleaning and correlation filtering reduce it to 17 features.
Verdict
A pre-trained Random Forest decides benign or malicious, offline, in under two seconds on a CPU. Benign files go straight to the report.
Explanation
For flagged files, the features become a search query. Retrieval pulls matching context, and a self-hosted Mistral 7B writes the category, ATT&CK mapping and next steps.
03 · Detection
An interpretable classifier.
The model was trained on 1,267 PDFs: 635 malicious samples from public malware-sharing sources, each verified with antivirus aggregators, and 632 benign documents from open repositories. Some benign files were deliberately chosen because they contain interactive elements, so the model couldn’t learn the shortcut “JavaScript means malicious”. All malware handling happened in an isolated VM with disposable snapshots.
Logistic regression, an SVM and a Random Forest were evaluated. The Random Forest won on robustness with mixed feature types, and because its feature importance can be read directly. It was tuned with 5-fold cross-validation on F1, using an 80/20 train/test split.
| Metric | Malicious | Benign | Macro avg. |
|---|---|---|---|
| Precision | 0.824 | 0.839 | 0.831 |
| Recall | 0.850 | 0.811 | 0.831 |
| F1 | 0.837 | 0.825 | 0.831 |
| Accuracy | — | — | 0.832 |
| ROC AUC | — | — | 0.928 |
The 17 features
- pdfid_URI
- num_objs
- streams_per_object
- file_entropy
- num_streams
- file_size
- num_urls
- pdfid_ObjStm
- has_acroform_marker
- pdfid_AcroForm
- pdfid_OpenAction
- pdfid_EmbeddedFile
- pdfid_AA
- pdfid_JS
- js_entropy_avg
- pdfid_Launch
- has_javascript_marker
Permutation importance, SHAP and LIME were used to check that the model leans on sensible signals. The top three were URI references, object count and streams per object.
04 · Explanation
From verdict to next step.
A verdict alone doesn’t help an analyst decide what to do. For every flagged file, PDFreak AI runs retrieval-augmented generation, entirely on its own hardware:
- Knowledge base: about 68,500 chunks, drawn from the malicious PDF corpus and from MITRE ATT&CK Enterprise techniques
- Retrieval: all-MiniLM-L6-v2 embeddings in a FAISS index, orchestrated with LangChain
- Model: Mistral 7B, served locally through Ollama, not fine-tuned
- Output: a threat category (phishing, JavaScript exploit, shellcode or embedded binary, obfuscated payload), an ATT&CK mapping, a plain-language summary and recommended actions
Example report · from the evaluation
verdict malicious
category JavaScript exploit
ATT&CK T1059.005
action monitor script execution events
Across 50 malicious files, 42 reports (84%) matched the true label, 45 (90%) referenced an appropriate ATT&CK technique, and 40 (80%) recommended mitigations consistent with standard playbooks.
05 · Built for hostile files
Assume every upload bites.
Everything PDFreak AI touches is, by design, possibly malicious. So the service around the model is hardened as carefully as the model itself:
- 10 MB upload limit and a 50 MB decompression cap, so archives can’t balloon
- Timeouts on every analysis step
- Zip Slip-safe extraction, locked-down working directories (
chmod 700) and an isolated temp area inside Docker - No external API calls: the classifier and the language model both run locally
The web app is React 18 and Tailwind on the front, FastAPI with OpenAPI docs behind it, and Docker Compose around both. An analyst drags in a file and watches it move through analysis, classification and report generation. The result is a verdict with a confidence score, the file’s structure and indicators, and the AI summary.


Six weeks of cloud infrastructure for the whole project cost £109.39. Commercial cloud antivirus subscriptions can exceed £1,000 a year.
06 · Honest limits
What’s next.
Limits today
- A small dataset (1,267 files) for a problem this varied
- No formal multi-class taxonomy, so AI labels were checked by hand
- The language model can still be generic, or wrong
- No study with working SOC analysts yet
- SHAP and LIME are too slow to run on every file
Next
- A larger dataset, and gradient-boosted or shallow neural models
- Fine-tuning the model on ATT&CK and CISA guidance
- On-demand explanations in the interface
- A usability study with analysts, JSON export and SIEM integration
- A fully containerised frontend for on-premises and air-gapped use
Get in touch.
Tell me what you run today and what you want to be true in three months. I will reply with whether I can help, and if not, who might.
- admin@thesefaway.com
There is no contact form on this site, so nothing you send is stored here. Email comes straight to me.