THESEFAWAY Products PDFreak AI

Research prototype · BSc Computer Science, Brunel University London · 2025

PDFreak AI.

Explainable PDF malware detection for small security teams. It tells an analyst whether a PDF is malicious, what kind of threat it is, which MITRE ATT&CK technique it maps to and what to do next, without sending the file to anyone else’s cloud.

  • Random Forest
  • Mistral 7B
  • FAISS retrieval
  • MITRE ATT&CK
  • Ghidra
  • FastAPI
  • React
  • Docker
Accuracy
83.2%
Random Forest, held-out test set
ROC AUC
0.928
How well it separates benign from malicious
ATT&CK mapping
90%
Of 50 reports cited an appropriate technique
Classifier time
< 2 s
Per file, on CPU, no GPU

01 · The problem

PDFs are an easy way in.

PDFs can carry JavaScript, embedded files, launch actions and obfuscated payloads, and they arrive by email every day. Signature-based antivirus catches known samples but misses new or disguised ones.

Large security teams answer this with commercial sandboxes and cloud AI services. Small SOC teams often can’t: the tools are expensive, and uploading a suspicious file, which may contain sensitive business data, to a third party is a privacy decision in its own right. They are also left with a verdict and no explanation.

PDFreak AI set out to answer three questions for a small team, on its own hardware: is this file malicious, what kind of threat is it, and what should I do about it?

02 · How it works

Classify first, explain second.

A fast, interpretable classifier handles every file. The language model, the expensive part, only runs on the files it flags.

Pipeline12 stages · 2 paths

PDFreak AI: full analysis pipelineAn uploaded PDF or ZIP is analysed statically (PDFiD, peepdf, pdf-parser) and dynamically (Ghidra, sandbox VM) in parallel. The outputs become one feature vector, reduced from over 45 raw features to 17, which a Random Forest classifies. Benign files go straight to the web report. Flagged files trigger retrieval over a knowledge base of MITRE ATT&CK and malicious-PDF chunks, and a self-hosted Mistral 7B writes the threat category, ATT&CK mapping and recommended actions into the report.flaggedbenigncontextexplainsupload · FastAPIPDF / ZIPPDFiD · peepdf · pdf-parserstatic analysisGhidra · sandbox VMdynamic analysis45+ raw → 17feature vectorunder 2 s on CPURandom ForestATT&CK + malwareknowledge baseMiniLM embeddingsFAISS retrievalOllama · self-hostedMistral 7Bverdict · ATT&CK · actionsweb reportstatic and dynamic analysis run in parallel · the language model only sees files the classifier flagsall models run self-hosted in Docker, with no external API calls
  1. Upload

    A PDF or ZIP goes to a FastAPI endpoint and lands in an isolated working directory inside Docker.

  2. Static analysis

    PDFiD, peepdf and pdf-parser pull out structure: objects, streams, scripts, and tags such as /OpenAction and /JavaScript.

  3. Dynamic analysis

    In parallel, Ghidra’s headless analyser looks for shellcode patterns, and a sandbox VM watches process creation, file and registry changes, and network activity.

  4. Features

    Over 45 raw signals become one vector, then cleaning and correlation filtering reduce it to 17 features.

  5. Verdict

    A pre-trained Random Forest decides benign or malicious, offline, in under two seconds on a CPU. Benign files go straight to the report.

  6. Explanation

    For flagged files, the features become a search query. Retrieval pulls matching context, and a self-hosted Mistral 7B writes the category, ATT&CK mapping and next steps.

03 · Detection

An interpretable classifier.

The model was trained on 1,267 PDFs: 635 malicious samples from public malware-sharing sources, each verified with antivirus aggregators, and 632 benign documents from open repositories. Some benign files were deliberately chosen because they contain interactive elements, so the model couldn’t learn the shortcut “JavaScript means malicious”. All malware handling happened in an isolated VM with disposable snapshots.

Logistic regression, an SVM and a Random Forest were evaluated. The Random Forest won on robustness with mixed feature types, and because its feature importance can be read directly. It was tuned with 5-fold cross-validation on F1, using an 80/20 train/test split.

Random Forest results on the held-out test set
MetricMaliciousBenignMacro avg.
Precision0.8240.8390.831
Recall0.8500.8110.831
F10.8370.8250.831
Accuracy——0.832
ROC AUC——0.928

The 17 features

  • pdfid_URI
  • num_objs
  • streams_per_object
  • file_entropy
  • num_streams
  • file_size
  • num_urls
  • pdfid_ObjStm
  • has_acroform_marker
  • pdfid_AcroForm
  • pdfid_OpenAction
  • pdfid_EmbeddedFile
  • pdfid_AA
  • pdfid_JS
  • js_entropy_avg
  • pdfid_Launch
  • has_javascript_marker

Permutation importance, SHAP and LIME were used to check that the model leans on sensible signals. The top three were URI references, object count and streams per object.

04 · Explanation

From verdict to next step.

A verdict alone doesn’t help an analyst decide what to do. For every flagged file, PDFreak AI runs retrieval-augmented generation, entirely on its own hardware:

  • Knowledge base: about 68,500 chunks, drawn from the malicious PDF corpus and from MITRE ATT&CK Enterprise techniques
  • Retrieval: all-MiniLM-L6-v2 embeddings in a FAISS index, orchestrated with LangChain
  • Model: Mistral 7B, served locally through Ollama, not fine-tuned
  • Output: a threat category (phishing, JavaScript exploit, shellcode or embedded binary, obfuscated payload), an ATT&CK mapping, a plain-language summary and recommended actions

Example report · from the evaluation

verdict malicious

category JavaScript exploit

ATT&CK T1059.005

action monitor script execution events

Matched the attack type, technique and a standard SOC playbook response.

Across 50 malicious files, 42 reports (84%) matched the true label, 45 (90%) referenced an appropriate ATT&CK technique, and 40 (80%) recommended mitigations consistent with standard playbooks.

05 · Built for hostile files

Assume every upload bites.

Everything PDFreak AI touches is, by design, possibly malicious. So the service around the model is hardened as carefully as the model itself:

  • 10 MB upload limit and a 50 MB decompression cap, so archives can’t balloon
  • Timeouts on every analysis step
  • Zip Slip-safe extraction, locked-down working directories (chmod 700) and an isolated temp area inside Docker
  • No external API calls: the classifier and the language model both run locally

The web app is React 18 and Tailwind on the front, FastAPI with OpenAPI docs behind it, and Docker Compose around both. An analyst drags in a file and watches it move through analysis, classification and report generation. The result is a verdict with a confidence score, the file’s structure and indicators, and the AI summary.

PDFreak AI upload screen: a drag-and-drop area, an Upload & Scan button and a verdict bar reading Benign PDF, confidence 64.55%.
Upload and scan.
PDFreak AI results: the verdict with its confidence score, a file overview with type, size and entropy, behavioural indicators showing shellcode patterns and URLs found, and quick stats for objects, streams and URLs.
The result view. The filename and hash of the test file are redacted.

Six weeks of cloud infrastructure for the whole project cost £109.39. Commercial cloud antivirus subscriptions can exceed £1,000 a year.

06 · Honest limits

What’s next.

Limits today

  • A small dataset (1,267 files) for a problem this varied
  • No formal multi-class taxonomy, so AI labels were checked by hand
  • The language model can still be generic, or wrong
  • No study with working SOC analysts yet
  • SHAP and LIME are too slow to run on every file

Next

  • A larger dataset, and gradient-boosted or shallow neural models
  • Fine-tuning the model on ATT&CK and CISA guidance
  • On-demand explanations in the interface
  • A usability study with analysts, JSON export and SIEM integration
  • A fully containerised frontend for on-premises and air-gapped use

← All products

Get in touch.

Tell me what you run today and what you want to be true in three months. I will reply with whether I can help, and if not, who might.

Email
admin@thesefaway.com

There is no contact form on this site, so nothing you send is stored here. Email comes straight to me.