同步完整源码 - 2026-05-25
This commit is contained in:
+21
@@ -0,0 +1,21 @@
|
||||
# Evitar subir archivos .zip
|
||||
/Salidas_archivos/*.zip
|
||||
|
||||
# Environment
|
||||
.venv/
|
||||
|
||||
# Configuracion
|
||||
config.json
|
||||
|
||||
# Cache
|
||||
__pycache__/
|
||||
|
||||
# Logs
|
||||
logs/
|
||||
*.log
|
||||
|
||||
#Tests
|
||||
.pytest_tmp/
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
Binary file not shown.
File diff suppressed because one or more lines are too long
@@ -0,0 +1,68 @@
|
||||
%PDF-1.3
|
||||
%“Œ‹ž ReportLab Generated PDF document http://www.reportlab.com
|
||||
1 0 obj
|
||||
<<
|
||||
/F1 2 0 R
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font
|
||||
>>
|
||||
endobj
|
||||
3 0 obj
|
||||
<<
|
||||
/Contents 7 0 R /MediaBox [ 0 0 612 792 ] /Parent 6 0 R /Resources <<
|
||||
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
|
||||
>> /Rotate 0 /Trans <<
|
||||
|
||||
>>
|
||||
/Type /Page
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/PageMode /UseNone /Pages 6 0 R /Type /Catalog
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/Author (anonymous) /CreationDate (D:20260203033950+00'00') /Creator (ReportLab PDF Library - www.reportlab.com) /Keywords () /ModDate (D:20260203033950+00'00') /Producer (ReportLab PDF Library - www.reportlab.com)
|
||||
/Subject (unspecified) /Title (untitled) /Trapped /False
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Count 1 /Kids [ 3 0 R ] /Type /Pages
|
||||
>>
|
||||
endobj
|
||||
7 0 obj
|
||||
<<
|
||||
/Filter [ /ASCII85Decode /FlateDecode ] /Length 126
|
||||
>>
|
||||
stream
|
||||
GapQh0E=F,0U\H3T\pNYT^QKk?tc>IP,;W#U1^23ihPEM_M(M8&8Hm/&J,g.K]FUr"Ltrj'PHPWAEG59Vrjfk/*JGskNCnVPb_?EjQgM0KOcNbBS!N@DufqXC*%!~>endstream
|
||||
endobj
|
||||
xref
|
||||
0 8
|
||||
0000000000 65535 f
|
||||
0000000073 00000 n
|
||||
0000000104 00000 n
|
||||
0000000211 00000 n
|
||||
0000000404 00000 n
|
||||
0000000472 00000 n
|
||||
0000000768 00000 n
|
||||
0000000827 00000 n
|
||||
trailer
|
||||
<<
|
||||
/ID
|
||||
[<2223e39864f5271a4fa9ba223bc69a84><2223e39864f5271a4fa9ba223bc69a84>]
|
||||
% ReportLab generated PDF document -- digest (http://www.reportlab.com)
|
||||
|
||||
/Info 5 0 R
|
||||
/Root 4 0 R
|
||||
/Size 8
|
||||
>>
|
||||
startxref
|
||||
1043
|
||||
%%EOF
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
this is not a valid pptx zip
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1 @@
|
||||
not a real xlsx
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,27 @@
|
||||
|
||||
---
|
||||
|
||||
## LICENSE
|
||||
|
||||
```txt
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 ING.Jose Leonardo Murillo Avalos
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,147 @@
|
||||
# Document-Image-Extractor
|
||||
|
||||
CLI tool to extract embedded images from **DOCX**, **PDF**, **PPTX** and **XLSX** files, with deduplication , size filtering, and batch export to ZIPs.
|
||||
|
||||
---
|
||||
|
||||
## Features:
|
||||
Extract images from:
|
||||
- DOCX (Word documents)
|
||||
- PDF (documents)
|
||||
- PPTX (Powerpoint documents)
|
||||
- XLSX (Excel documents)
|
||||
|
||||
Outputs:
|
||||
- Creates a **ZIP per input file** with extracted images
|
||||
|
||||
built-in helpers:
|
||||
- **Deduplication** (skips repeated images within the same document)
|
||||
- **Size filter** (`min_kb` default is 5kb)
|
||||
- Handles “no images” and **corrupt files** gracefully
|
||||
|
||||
---
|
||||
|
||||
## Project status
|
||||
this repository is begin improved **phase by phase**
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
- python 3.12+ (recomended)
|
||||
|
||||
Dependencies (install from 'requirements.txt'):
|
||||
- python-docx
|
||||
- PyMuPDF
|
||||
- pillow
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Clone the repository
|
||||
```bash
|
||||
git clone https://github.com/LeoMurilloDev/document-image-extractor.git
|
||||
cd document-image-extractor
|
||||
```
|
||||
|
||||
### 2. Create and activate a virtual environment
|
||||
#### Windows
|
||||
```bash
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\activate
|
||||
```
|
||||
#### macOS / Linux
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### 3. Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
## Usage
|
||||
|
||||
### Folder structure expected by the script
|
||||
the script creates these folders automatically if they don't exist:
|
||||
- **Entrdas_archivos/** -> place your **.docx** and **.pdf** files here
|
||||
- **Salidas_archivos/** -> output ZIPs will be generated here
|
||||
- **temp/** -> temporary extraction folder (auto-cleaned)
|
||||
|
||||
### Configuration
|
||||
You can customize filters without editing the code using `config.json` (repo root).
|
||||
Example:
|
||||
```json
|
||||
{
|
||||
"filters": {
|
||||
"min_kb": 5,
|
||||
"min_width": 0,
|
||||
"min_height": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
- `min_kb`: minimum file size in kb (default: 5)
|
||||
- `min_width`/ `min_height`: optional dimension filter (0 disables it)
|
||||
|
||||
|
||||
## Run
|
||||
```bash
|
||||
python main.py
|
||||
```
|
||||
|
||||
## CLI usage
|
||||
The tool can be used with default folders/config:
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
|
||||
python main.py --input Entradas_archivos --output Salidas_archivos
|
||||
|
||||
python main.py --input example.pptx --output Salidas_archivos
|
||||
|
||||
python main.py --input Entradas_archivos --recursive
|
||||
|
||||
python main.py --input Entradas_archivos --min-kb 1 --min-width 100 --min-height 100
|
||||
|
||||
python main.py --input Entradas_archivos --no-dedup
|
||||
|
||||
python main.py --input Entradas_archivos --format folder
|
||||
|
||||
python main.py --input Entradas_archivos --log-level DEBUG --log-file logs/debug.log
|
||||
```
|
||||
|
||||
## Output
|
||||
- For each input file, a ZIP is created in **Salidas_archivos/**
|
||||
- Example:
|
||||
- Input: **Entradas_archivos/report.pdf**
|
||||
- Output: **Salidas_archivos/report.zip**
|
||||
|
||||
## What to expect
|
||||
When you run the script, it prints a summary per file:
|
||||
- `guardadas` -> images saved successfully
|
||||
- `duplicadas` -> images skipped due to hash duplication
|
||||
- `pequeñas` -> images filtered out by size
|
||||
- `encontradas` -> images found inside the document
|
||||
### Important notes
|
||||
- In `DOCX`, images are saved using the real extension (.jpg, .png, .gif, etc)
|
||||
- `temp/` is cleaned even when a file fails
|
||||
|
||||
## Test suites
|
||||
we use small test suites to validate.
|
||||
### Documents to try
|
||||
Includes:
|
||||
- Mixed formats (JPG/PNG/GIF)
|
||||
- Duplicates
|
||||
- Small icon filtered out by size
|
||||
- Corrupt files (error handling)
|
||||
Manual validation steps:
|
||||
1. Copy test files into `Entradas_archivos/`
|
||||
2. Run `python main.py`
|
||||
3. Verify
|
||||
- Output ZIPs exist in `Salidas_archivos/`
|
||||
- Extencions are correct in DOCX resutls (.jpg, .png, .gif)
|
||||
- Duplicates are removed
|
||||
- `temp/` is empty at the end
|
||||
|
||||
## Contributing
|
||||
if you want to propose changes:
|
||||
1. Fork the repo
|
||||
2. Create a branch
|
||||
3. Open a PR with a clear description
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"paths": {
|
||||
"input_dir": "Entradas_archivos",
|
||||
"output_dir": "Salidas_archivos",
|
||||
"temp_dir": "temp"
|
||||
},
|
||||
"filters": {
|
||||
"min_kb": 5,
|
||||
"min_width": 0,
|
||||
"min_height": 0
|
||||
},
|
||||
"dedup": {"enabled": true},
|
||||
"output": {"format": "zip"},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"log_file": "logs/run.log",
|
||||
"format": "%(asctime)s | %(levelname)s | %(message)s",
|
||||
"datefmt": "%Y-%m-%d %H:%M:%S"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
__version__ = "0.1"
|
||||
__all__ = ["config", "pipeline", "report"]
|
||||
@@ -0,0 +1,162 @@
|
||||
import argparse
|
||||
from typing import Any, Dict
|
||||
from . import __version__
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="document-image-extractor",
|
||||
description=(
|
||||
"Extract embedded images from PDF, DOCX, PPTX and XLSX files. "
|
||||
"Supports deduplication, filters and ZIP/folder output."
|
||||
),
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"-c",
|
||||
"--config",
|
||||
dest="config_path",
|
||||
default="config.json",
|
||||
help="Ruta del archivo de configuración. Default: config.json",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"-i",
|
||||
"--input",
|
||||
dest="input_path",
|
||||
default=None,
|
||||
help="Archivo o carpeta de entrada. Default: paths.input_dir del config.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"-o",
|
||||
"--output",
|
||||
dest="output_dir",
|
||||
default=None,
|
||||
help="Carpeta de salida. Default: paths.output_dir del config.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--temp-dir",
|
||||
dest="temp_dir",
|
||||
default=None,
|
||||
help="Carpeta temporal. Default: paths.temp_dir del config.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"-r",
|
||||
"--recursive",
|
||||
action="store_true",
|
||||
help="Procesa archivos dentro de subcarpetas cuando --input es una carpeta.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--min-kb",
|
||||
dest="min_kb",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Tamaño mínimo en KB para guardar una imagen.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--min-width",
|
||||
dest="min_width",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Ancho mínimo en pixeles. 0 desactiva el filtro.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--min-height",
|
||||
dest="min_height",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Alto mínimo en pixeles. 0 desactiva el filtro.",
|
||||
)
|
||||
|
||||
dedup_group = parser.add_mutually_exclusive_group()
|
||||
dedup_group.add_argument(
|
||||
"--dedup",
|
||||
dest="dedup_enabled",
|
||||
action="store_true",
|
||||
help="Activa deduplicación por hash.",
|
||||
)
|
||||
dedup_group.add_argument(
|
||||
"--no-dedup",
|
||||
dest="dedup_enabled",
|
||||
action="store_false",
|
||||
help="Desactiva deduplicación por hash.",
|
||||
)
|
||||
parser.set_defaults(dedup_enabled=None)
|
||||
|
||||
parser.add_argument(
|
||||
"--format",
|
||||
dest="output_format",
|
||||
choices=["zip", "folder"],
|
||||
default=None,
|
||||
help="Formato de salida. Opciones: zip, folder. Default: config output.format.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--log-level",
|
||||
dest="log_level",
|
||||
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
|
||||
default=None,
|
||||
help="Nivel de logs.",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--log-file",
|
||||
dest="log_file",
|
||||
default=None,
|
||||
help="Ruta del archivo de logs. Ejemplo: logs/run.log",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--version",
|
||||
action="version",
|
||||
version=f"%(prog)s {__version__}",
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
def apply_cli_overrides(cfg: Dict[str, Any], args: argparse.Namespace) -> Dict[str, Any]:
|
||||
cfg.setdefault("runtime", {})
|
||||
|
||||
# Input runtime: puede ser archivo o carpeta
|
||||
cfg["runtime"]["input_path"] = args.input_path or cfg["paths"]["input_dir"]
|
||||
cfg["runtime"]["input_was_provided"] = bool(args.input_path)
|
||||
cfg["runtime"]["recursive"] = bool(args.recursive)
|
||||
|
||||
# Paths
|
||||
if args.output_dir is not None:
|
||||
cfg["paths"]["output_dir"] = args.output_dir
|
||||
|
||||
if args.temp_dir is not None:
|
||||
cfg["paths"]["temp_dir"] = args.temp_dir
|
||||
|
||||
# Filters
|
||||
if args.min_kb is not None:
|
||||
cfg["filters"]["min_kb"] = int(args.min_kb)
|
||||
|
||||
if args.min_width is not None:
|
||||
cfg["filters"]["min_width"] = int(args.min_width)
|
||||
|
||||
if args.min_height is not None:
|
||||
cfg["filters"]["min_height"] = int(args.min_height)
|
||||
|
||||
# Dedup
|
||||
if args.dedup_enabled is not None:
|
||||
cfg["dedup"]["enabled"] = bool(args.dedup_enabled)
|
||||
|
||||
# Output
|
||||
if args.output_format is not None:
|
||||
cfg["output"]["format"] = args.output_format
|
||||
|
||||
# Logging
|
||||
if args.log_level is not None:
|
||||
cfg["logging"]["level"] = args.log_level
|
||||
|
||||
if args.log_file is not None:
|
||||
cfg["logging"]["log_file"] = args.log_file
|
||||
|
||||
return cfg
|
||||
@@ -0,0 +1,54 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
DEFAULT_CONFIG: Dict[str, Any] = {
|
||||
"paths": {
|
||||
"input_dir": "Entradas_archivos",
|
||||
"output_dir": "Salidas_archivos",
|
||||
"temp_dir": "temp",
|
||||
},
|
||||
"filters": {
|
||||
"min_kb": 5,
|
||||
"min_width": 0,
|
||||
"min_height": 0,
|
||||
},
|
||||
"dedup": {"enabled": True},
|
||||
"output": {"format": "zip"},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"log_file": "logs/run.log",
|
||||
"format": "%(asctime)s | %(levelname)s | %(message)s",
|
||||
"datefmt": "%Y-%m-%d %H:%M:%S"
|
||||
}
|
||||
}
|
||||
|
||||
# Cargar la configuracion del programa y parametros basicos
|
||||
def load_config(config_path: Path = Path("config.json")) -> Dict[str, Any]:
|
||||
# Carga de archivo config.json y en caso de configuraciones faltantes he hace mescla con DEFAULT_CONFIG
|
||||
cfg = json.loads(json.dumps(DEFAULT_CONFIG))
|
||||
if not config_path.exists(): return cfg
|
||||
|
||||
try:
|
||||
user_cfg = json.loads(config_path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return cfg
|
||||
|
||||
for section in ("paths", "filters", "dedup", "output", "logging"):
|
||||
if isinstance(user_cfg.get(section), dict): cfg[section].update(user_cfg[section])
|
||||
|
||||
# Normalizacion minima
|
||||
cfg["filters"]["min_kb"] = int(cfg["filters"].get("min_kb", 5))
|
||||
cfg["filters"]["min_width"] = int(cfg["filters"].get("min_width", 0))
|
||||
cfg["filters"]["min_height"] = int(cfg["filters"].get("min_height", 0))
|
||||
cfg["dedup"]["enabled"] = bool(cfg["dedup"].get("enabled", True))
|
||||
cfg["output"]["format"] = str(cfg["output"].get("format", "zip")).lower()
|
||||
cfg["logging"]["level"] = str(cfg["logging"].get("level", "INFO")).upper()
|
||||
cfg["logging"]["format"] = str(cfg["logging"].get("format", DEFAULT_CONFIG["logging"]["format"]))
|
||||
cfg["logging"]["datefmt"] = str(cfg["logging"].get("datefmt", DEFAULT_CONFIG["logging"]["datefmt"]))
|
||||
log_file = str(cfg["logging"].get("log_file", "")).strip()
|
||||
if not log_file:
|
||||
log_file = DEFAULT_CONFIG["logging"]["log_file"]
|
||||
cfg["logging"]["log_file"] = log_file
|
||||
|
||||
return cfg
|
||||
@@ -0,0 +1 @@
|
||||
__all__ = ["docx_extractor", "pdf_extractor","pptx_extractor","xlsx_extractor","ooxml_media_extractor"]
|
||||
@@ -0,0 +1,70 @@
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Set
|
||||
from docx import Document
|
||||
from ..utils.hashing import md5_bytes
|
||||
from ..utils.files import is_small_kb
|
||||
from ..utils.images import get_image_size, fails_dimension_filter, normalize_ext
|
||||
|
||||
|
||||
def _docx_image_ext(image_part) -> str:
|
||||
ct = getattr(image_part, "content_type", None)
|
||||
if ct and "/" in ct:
|
||||
return normalize_ext(ct.split("/")[-1])
|
||||
|
||||
partname = str(getattr(image_part, "partname", "")).lower()
|
||||
if "." in partname:
|
||||
return normalize_ext(partname.split(".")[-1])
|
||||
|
||||
return "bin"
|
||||
|
||||
|
||||
def extract_docx_images(docx_path: Path, temp_folder: Path, cfg: Dict[str, Any]) -> Dict[str, int]:
|
||||
filters = cfg["filters"]
|
||||
min_kb = int(filters["min_kb"])
|
||||
min_w = int(filters["min_width"])
|
||||
min_h = int(filters["min_height"])
|
||||
dedup_enabled = bool(cfg["dedup"]["enabled"])
|
||||
stats: Dict[str, int] = {"found": 0, "saved": 0, "duplicates": 0, "filtered_small": 0, "filtered_dims": 0, "errors": 0}
|
||||
|
||||
doc = Document(str(docx_path))
|
||||
image_parts = list(doc.part.package.image_parts)
|
||||
stats["found"] = len(image_parts)
|
||||
hashes: Set[str] = set()
|
||||
|
||||
for i, image_part in enumerate(image_parts, start=1):
|
||||
out_path = None
|
||||
try:
|
||||
blob = image_part.blob
|
||||
digest = md5_bytes(blob)
|
||||
|
||||
if dedup_enabled and digest in hashes:
|
||||
stats["duplicates"] += 1
|
||||
continue
|
||||
|
||||
ext = _docx_image_ext(image_part)
|
||||
out_path = temp_folder / f"image_{i:03d}.{ext}"
|
||||
out_path.write_bytes(blob)
|
||||
|
||||
if is_small_kb(out_path, min_kb):
|
||||
stats["filtered_small"] += 1
|
||||
out_path.unlink(missing_ok=True)
|
||||
continue
|
||||
|
||||
dims = get_image_size(out_path)
|
||||
if fails_dimension_filter(dims, min_w, min_h):
|
||||
stats["filtered_dims"] += 1
|
||||
out_path.unlink(missing_ok=True)
|
||||
continue
|
||||
|
||||
if dedup_enabled:
|
||||
hashes.add(digest)
|
||||
|
||||
stats["saved"] += 1
|
||||
except Exception:
|
||||
stats["errors"] += 1
|
||||
if out_path is not None:
|
||||
try:
|
||||
out_path.unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
return stats
|
||||
@@ -0,0 +1,67 @@
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, Set, Tuple
|
||||
from ..utils.hashing import md5_bytes
|
||||
from ..utils.files import is_small_kb
|
||||
from ..utils.images import get_image_size, fails_dimension_filter, normalize_ext
|
||||
|
||||
IMAGE_EXTS = {"png", "jpg", "jpeg", "gif", "bmp", "tiff", "tif", "webp", "emf", "wmf", "svg"}
|
||||
|
||||
def extract_ooxml_media_images(archive_path: Path, media_prefixes: Tuple[str, ...], temp_folder: Path, cfg: Dict[str, Any], source_tag: str) -> Dict[str, int]:
|
||||
# Extraer imagenes desde un OOXML (pptx, xlsx) leyendo lor archivos en media/
|
||||
|
||||
filters = cfg["filters"]
|
||||
min_kb = int(filters["min_kb"])
|
||||
min_w = int(filters["min_width"])
|
||||
min_h = int(filters["min_height"])
|
||||
dedup_enabled = bool(cfg["dedup"]["enabled"])
|
||||
|
||||
stats: Dict[str, int] = {"found": 0, "saved": 0, "duplicates": 0, "filtered_small": 0, "filtered_dims": 0, "errors": 0}
|
||||
hashes: Set[str] = set()
|
||||
counter = 0
|
||||
|
||||
with zipfile.ZipFile(archive_path, "r") as zf:
|
||||
for info in zf.infolist():
|
||||
name = info.filename
|
||||
|
||||
if name.endswith("/"): continue
|
||||
if not any(name.startswith(prefix) for prefix in media_prefixes): continue
|
||||
ext = normalize_ext(Path(name).suffix.replace(".",""))
|
||||
if not ext or ext not in IMAGE_EXTS: continue
|
||||
|
||||
out_path = None
|
||||
try:
|
||||
data = zf.read(name)
|
||||
stats["found"] += 1
|
||||
|
||||
digest = md5_bytes(data)
|
||||
if dedup_enabled and digest in hashes:
|
||||
stats["duplicates"] += 1
|
||||
continue
|
||||
|
||||
counter += 1
|
||||
out_path = temp_folder / f"image_{counter:03d}_{source_tag}.{ext}"
|
||||
out_path.write_bytes(data)
|
||||
|
||||
if is_small_kb(out_path, min_kb):
|
||||
stats["filtered_small"] += 1
|
||||
out_path.unlink(missing_ok=True)
|
||||
continue
|
||||
|
||||
dims = get_image_size(out_path)
|
||||
if fails_dimension_filter(dims, min_w, min_h):
|
||||
stats["filtered_dims"] += 1
|
||||
out_path.unlink(missing_ok=True)
|
||||
continue
|
||||
|
||||
if dedup_enabled: hashes.add(digest)
|
||||
|
||||
stats["saved"] += 1
|
||||
except Exception:
|
||||
stats["errors"] += 1
|
||||
if out_path is not None:
|
||||
try:
|
||||
out_path.unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
return stats
|
||||
@@ -0,0 +1,68 @@
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Set
|
||||
import fitz
|
||||
from ..utils.hashing import md5_bytes
|
||||
from ..utils.files import is_small_kb
|
||||
from ..utils.images import get_image_size, fails_dimension_filter, normalize_ext
|
||||
|
||||
def extract_pdf_images(pdf_path: Path, temp_folder: Path, cfg: Dict[str, Any]) -> Dict[str, int]:
|
||||
# Extraer las imagenes de vienen en el documento pdf
|
||||
filters = cfg["filters"]
|
||||
min_kb = filters["min_kb"]
|
||||
min_w = filters["min_width"]
|
||||
min_h = filters["min_height"]
|
||||
dedup_enabled = cfg["dedup"]["enabled"]
|
||||
|
||||
stats = {"found": 0,"saved": 0, "duplicates": 0, "filtered_small": 0, "filtered_dims": 0, "errors": 0}
|
||||
|
||||
hashes: Set[str] = set()
|
||||
counter = 0
|
||||
|
||||
with fitz.open(str(pdf_path)) as pdf:
|
||||
for page_index in range(len(pdf)):
|
||||
page = pdf[page_index]
|
||||
for img in page.get_images(full=True):
|
||||
out_path = None
|
||||
try:
|
||||
xref = img[0]
|
||||
base = pdf.extract_image(xref)
|
||||
data = base.get("image", b"")
|
||||
ext = normalize_ext(base.get("ext", "bin"))
|
||||
|
||||
stats["found"] += 1
|
||||
|
||||
digest = md5_bytes(data)
|
||||
if dedup_enabled and digest in hashes:
|
||||
stats["duplicates"] += 1
|
||||
continue
|
||||
|
||||
counter += 1
|
||||
out_path = temp_folder / f"image_{counter:03d}_p{page_index+1:03d}.{ext}"
|
||||
out_path.write_bytes(data)
|
||||
|
||||
# Filtro KB
|
||||
if is_small_kb(out_path, min_kb):
|
||||
stats["filtered_small"] += 1
|
||||
out_path.unlink(missing_ok=True)
|
||||
continue
|
||||
|
||||
# Filtro de dimenciones
|
||||
dims = get_image_size(out_path)
|
||||
if fails_dimension_filter(dims, min_w, min_h):
|
||||
stats["filtered_dims"] += 1
|
||||
out_path.unlink(missing_ok=True)
|
||||
continue
|
||||
|
||||
if dedup_enabled:
|
||||
hashes.add(digest)
|
||||
|
||||
stats["saved"] += 1
|
||||
|
||||
except Exception:
|
||||
stats["errors"] += 1
|
||||
if out_path is not None:
|
||||
try:
|
||||
out_path.unlink(missing_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
return stats
|
||||
@@ -0,0 +1,12 @@
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
from .ooxml_media_extractor import extract_ooxml_media_images
|
||||
|
||||
def extract_pptx_images(pptx_path: Path, temp_folder: Path, cfg: Dict[str, Any]) -> Dict[str, int]:
|
||||
return extract_ooxml_media_images(archive_path=pptx_path,
|
||||
media_prefixes=("ppt/media/",),
|
||||
temp_folder=temp_folder,
|
||||
cfg=cfg,
|
||||
source_tag="pptx")
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
from .ooxml_media_extractor import extract_ooxml_media_images
|
||||
|
||||
def extract_xlsx_images(xlsx_path: Path, temp_folder: Path, cfg: Dict[str, Any]) -> Dict[str, int]:
|
||||
return extract_ooxml_media_images(
|
||||
archive_path=xlsx_path,
|
||||
media_prefixes=("xl/media/",),
|
||||
temp_folder=temp_folder,
|
||||
cfg=cfg,
|
||||
source_tag="xlsx",
|
||||
)
|
||||
@@ -0,0 +1,20 @@
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
def setup_logging(cfg: Dict[str, Any]) -> None:
|
||||
log_cfg = cfg.get("logging", {}) if isinstance(cfg, dict) else {}
|
||||
|
||||
level_name = str(log_cfg.get("level", "INFO")).upper()
|
||||
level = getattr(logging, level_name, logging.INFO)
|
||||
|
||||
fmt = str(log_cfg.get("format", "%(asctime)s | %(levelname)s | %(message)s"))
|
||||
datefmt = str(log_cfg.get("datefmt", "%Y-%m-%d %H:%M:%S"))
|
||||
|
||||
log_file = str(log_cfg.get("log_file", "logs/run.log")).strip() or "logs/run.log"
|
||||
log_path = Path(log_file)
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
handlers = [logging.StreamHandler(), logging.FileHandler(log_path, encoding="utf-8")]
|
||||
|
||||
logging.basicConfig(level=level, format=fmt, datefmt=datefmt, handlers=handlers, force=True)
|
||||
@@ -0,0 +1,74 @@
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
from .extractors.docx_extractor import extract_docx_images
|
||||
from .extractors.pdf_extractor import extract_pdf_images
|
||||
from .extractors.pptx_extractor import extract_pptx_images
|
||||
from .extractors.xlsx_extractor import extract_xlsx_images
|
||||
from .utils.files import clean_dir, ensure_dir, create_zip_from_folder, copy_folder
|
||||
|
||||
SUPPORTED_EXTS = {".pdf", ".docx", ".pptx", ".xlsx"}
|
||||
|
||||
def process_file(file_path: Path, cfg: Dict[str, Any]) -> Dict:
|
||||
paths = cfg["paths"]
|
||||
temp_root = Path(paths["temp_dir"])
|
||||
output_dir = Path(paths["output_dir"])
|
||||
output_format = cfg["output"]["format"]
|
||||
|
||||
ensure_dir(temp_root)
|
||||
ensure_dir(output_dir)
|
||||
|
||||
name = file_path.stem
|
||||
temp_folder = temp_root / name
|
||||
|
||||
clean_dir(temp_folder)
|
||||
ensure_dir(temp_folder)
|
||||
|
||||
try:
|
||||
ext = file_path.suffix.lower()
|
||||
|
||||
if ext == ".pdf":
|
||||
stats = extract_pdf_images(file_path, temp_folder, cfg)
|
||||
elif ext == ".docx":
|
||||
stats = extract_docx_images(file_path, temp_folder, cfg)
|
||||
elif ext == ".pptx":
|
||||
stats = extract_pptx_images(file_path, temp_folder, cfg)
|
||||
elif ext == ".xlsx":
|
||||
stats = extract_xlsx_images(file_path, temp_folder, cfg)
|
||||
else:
|
||||
return {"skipped": True, "reason": f"extencion no soportada ({ext})"}
|
||||
|
||||
if stats["saved"] > 0:
|
||||
if output_format == "zip":
|
||||
zip_path = output_dir / f"{name}.zip"
|
||||
create_zip_from_folder(temp_folder, zip_path)
|
||||
|
||||
elif output_format == "folder":
|
||||
folder_path = output_dir / name
|
||||
copy_folder(temp_folder, folder_path)
|
||||
|
||||
else:
|
||||
return {"error": f"formato de salida no soportado: {output_format}"}
|
||||
|
||||
return stats
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
finally:
|
||||
clean_dir(temp_folder)
|
||||
|
||||
def list_input_files(input_path: Path, recursive: bool = False) -> List[Path]:
|
||||
if input_path.is_file():
|
||||
if input_path.suffix.lower() in SUPPORTED_EXTS:
|
||||
return [input_path]
|
||||
return []
|
||||
|
||||
if not input_path.is_dir():
|
||||
return []
|
||||
|
||||
pattern = "**/*" if recursive else "*"
|
||||
|
||||
files = [
|
||||
f for f in input_path.glob(pattern)
|
||||
if f.is_file() and f.suffix.lower() in SUPPORTED_EXTS
|
||||
]
|
||||
|
||||
return sorted(files)
|
||||
@@ -0,0 +1,52 @@
|
||||
import logging
|
||||
from typing import Dict
|
||||
logger = logging.getLogger("document_image_extractor")
|
||||
|
||||
|
||||
def log_file_report(filename: str, stats: Dict) -> None:
|
||||
if stats.get("skipped"):
|
||||
logger.warning("%s: %s", filename, stats.get("reason"))
|
||||
return
|
||||
|
||||
if "error" in stats:
|
||||
logger.error("%s: %s", filename, stats["error"])
|
||||
return
|
||||
|
||||
if stats.get("saved", 0) > 0:
|
||||
logger.info(
|
||||
"%s | found=%s, saved=%s, dupes=%s, filtered_small=%s, filtered_dims=%s, errors=%s",
|
||||
filename,
|
||||
stats.get("found", 0),
|
||||
stats.get("saved", 0),
|
||||
stats.get("duplicates", 0),
|
||||
stats.get("filtered_small", 0),
|
||||
stats.get("filtered_dims", 0),
|
||||
stats.get("errors", 0),
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"%s | No hay imágenes válidas | found=%s, dupes=%s, filtered_small=%s, filtered_dims=%s, errors=%s",
|
||||
filename,
|
||||
stats.get("found", 0),
|
||||
stats.get("duplicates", 0),
|
||||
stats.get("filtered_small", 0),
|
||||
stats.get("filtered_dims", 0),
|
||||
stats.get("errors", 0),
|
||||
)
|
||||
|
||||
def accumulate_totals(total: Dict[str, int], stats: Dict) -> None:
|
||||
for k in total:
|
||||
total[k] += int(stats.get(k, 0))
|
||||
|
||||
def log_summary(files: int, skipped: int, failed: int, total: Dict[str, int]) -> None:
|
||||
logger.info("=== RESUMEN ===")
|
||||
logger.info(
|
||||
"files=%s, skipped=%s, failed=%s | found=%s, saved=%s, dupes=%s, filtered_small=%s, filtered_dims=%s, errors=%s",
|
||||
files, skipped, failed,
|
||||
total.get("found", 0),
|
||||
total.get("saved", 0),
|
||||
total.get("duplicates", 0),
|
||||
total.get("filtered_small", 0),
|
||||
total.get("filtered_dims", 0),
|
||||
total.get("errors", 0),
|
||||
)
|
||||
@@ -0,0 +1 @@
|
||||
__all_ = ["files", "hashing", "images"]
|
||||
@@ -0,0 +1,27 @@
|
||||
import shutil
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
def ensure_dir(path: Path) -> None:
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def clean_dir(path: Path) -> None:
|
||||
# Eliminar archivos de la carpeta esfecifica
|
||||
if path.exists(): shutil.rmtree(path, ignore_errors=True)
|
||||
|
||||
def create_zip_from_folder(folder: Path, zip_destine: Path) -> None:
|
||||
# Crear un zip con el contenido de folder
|
||||
with zipfile.ZipFile(zip_destine, 'w', compression=zipfile.ZIP_DEFLATED) as zf:
|
||||
for file in folder.rglob("*"):
|
||||
if file.is_file(): zf.write(file, arcname=file.relative_to(folder))
|
||||
|
||||
def is_small_kb(path: Path, min_kb: int) -> bool:
|
||||
# Validacion booleana si la imagen es mas chica que el parametro min_kb configurado
|
||||
return (path.stat().st_size / 1024) < min_kb
|
||||
|
||||
def copy_folder(src: Path, dest: Path) -> None:
|
||||
# Copia una carpeta al destino, si el destino existe lo remplaza
|
||||
if dest.exists(): shutil.rmtree(dest, ignore_errors=True)
|
||||
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copytree(src, dest)
|
||||
@@ -0,0 +1,4 @@
|
||||
import hashlib
|
||||
|
||||
def md5_bytes(data: bytes) -> str:
|
||||
return hashlib.md5(data).hexdigest()
|
||||
@@ -0,0 +1,32 @@
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
from PIL import Image
|
||||
|
||||
|
||||
def get_image_size(path: Path) -> Optional[Tuple[int, int]]:
|
||||
# Devolver el ancho y el alto de las imagenes, En caso de no poder abrir regresa None
|
||||
try:
|
||||
with Image.open(path) as im:
|
||||
return im.size
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def fails_dimension_filter(dims: Optional[Tuple[int, int]], min_w: int, min_h: int) -> bool:
|
||||
# En caso de que dims sea None se omiten validaciones
|
||||
if min_w <= 0 and min_h <= 0: return False
|
||||
if dims is None: return False
|
||||
|
||||
w, h = dims
|
||||
if min_w > 0 and w < min_w: return True
|
||||
if min_h > 0 and h < min_h: return True
|
||||
|
||||
return False
|
||||
|
||||
def normalize_ext(ext: str) -> str:
|
||||
# Normalizacion de extenciones para imagenes
|
||||
ext = (ext or "bin").lower().strip(".")
|
||||
if ext in ("jpeg", "jpe"):
|
||||
return "jpg"
|
||||
if ext == "tif":
|
||||
return "tiff"
|
||||
return ext
|
||||
@@ -0,0 +1,80 @@
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from document_image_extractor.cli import parse_args, apply_cli_overrides
|
||||
from document_image_extractor.config import load_config
|
||||
from document_image_extractor.logging_setup import setup_logging
|
||||
from document_image_extractor.pipeline import list_input_files, process_file, SUPPORTED_EXTS
|
||||
from document_image_extractor.report import log_file_report, accumulate_totals, log_summary
|
||||
from document_image_extractor.utils.files import ensure_dir
|
||||
|
||||
logger = logging.getLogger("document_image_extractor")
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
cfg = load_config(Path(args.config_path))
|
||||
cfg = apply_cli_overrides(cfg, args)
|
||||
setup_logging(cfg)
|
||||
paths = cfg["paths"]
|
||||
runtime = cfg["runtime"]
|
||||
|
||||
input_path = Path(runtime["input_path"])
|
||||
output_dir = Path(paths["output_dir"])
|
||||
temp_dir = Path(paths["temp_dir"])
|
||||
recursive = bool(runtime["recursive"])
|
||||
|
||||
if not input_path.exists():
|
||||
if runtime.get("input_was_provided"):
|
||||
logger.error("La ruta de entrada no existe: %s", input_path)
|
||||
return
|
||||
ensure_dir(input_path)
|
||||
|
||||
ensure_dir(output_dir)
|
||||
ensure_dir(temp_dir)
|
||||
|
||||
logger.debug("Input path: %s", input_path.resolve())
|
||||
logger.debug("Output dir: %s", output_dir.resolve())
|
||||
logger.debug("Temp dir: %s", temp_dir.resolve())
|
||||
logger.debug("Recursive: %s", recursive)
|
||||
logger.debug("Supported extensions: %s", ", ".join(sorted(SUPPORTED_EXTS)))
|
||||
logger.debug(
|
||||
"Filters: min_kb=%s, min_width=%s, min_height=%s",
|
||||
cfg["filters"]["min_kb"],
|
||||
cfg["filters"]["min_width"],
|
||||
cfg["filters"]["min_height"],
|
||||
)
|
||||
logger.debug("Output format: %s", cfg["output"]["format"])
|
||||
logger.debug("Dedup enabled: %s", cfg["dedup"]["enabled"])
|
||||
|
||||
files = list_input_files(input_path, recursive=recursive)
|
||||
|
||||
if not files:
|
||||
logger.warning(
|
||||
"No se encontraron archivos soportados (%s) en: %s",
|
||||
", ".join(sorted(SUPPORTED_EXTS)),
|
||||
input_path.resolve(),
|
||||
)
|
||||
return
|
||||
|
||||
total = {"found": 0, "saved": 0, "duplicates": 0, "filtered_small": 0, "filtered_dims": 0, "errors": 0}
|
||||
skipped = 0
|
||||
failed = 0
|
||||
|
||||
for f in files:
|
||||
stats = process_file(f, cfg)
|
||||
log_file_report(f.name, stats)
|
||||
|
||||
if stats.get("skipped"):
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
if "error" in stats:
|
||||
failed += 1
|
||||
continue
|
||||
|
||||
accumulate_totals(total, stats)
|
||||
|
||||
log_summary(files=len(files), skipped=skipped, failed=failed, total=total)
|
||||
|
||||
if __name__== "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,4 @@
|
||||
[pytest]
|
||||
testpaths = tests
|
||||
python_files = test_*.py
|
||||
addopts = -ra --basetemp=.pytest_tmp
|
||||
@@ -0,0 +1 @@
|
||||
pytest==8.0
|
||||
@@ -0,0 +1,3 @@
|
||||
python-docx==1.2.0
|
||||
PyMuPDF==1.26.7
|
||||
pillow==12.1.0
|
||||
@@ -0,0 +1,132 @@
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
import fitz
|
||||
import pytest
|
||||
from docx import Document
|
||||
from PIL import Image, ImageDraw
|
||||
|
||||
@pytest.fixture
|
||||
def sample_images(tmp_path: Path) -> Dict[str, Path]:
|
||||
assets = tmp_path / "assets"
|
||||
assets.mkdir()
|
||||
|
||||
big_png = assets / "big_simple.png"
|
||||
small_icon = assets / "small_icon.png"
|
||||
photo_jpg = assets / "photo.jpg"
|
||||
|
||||
# Imagen grande y comprimible
|
||||
img = Image.new("RGB", (900, 600), (245, 245, 245))
|
||||
draw = ImageDraw.Draw(img)
|
||||
draw.rectangle([40, 40, 860, 560], outline=(0, 0, 0), width=6)
|
||||
draw.text((60, 60), "BIG PNG", fill=(0, 0, 0))
|
||||
img.save(big_png, format="PNG", optimize=True)
|
||||
|
||||
# Icono Pequeño
|
||||
icon = Image.new("RGB", (32, 32), (255, 255, 255))
|
||||
draw = ImageDraw.Draw(icon)
|
||||
draw.rectangle([2, 2, 29, 29], outline=(0, 0, 0), width=2)
|
||||
draw.line([6, 16, 26, 16], fill=(0, 0, 0), width=2)
|
||||
draw.line([16, 6, 16, 26], fill=(0, 0, 0), width=2)
|
||||
icon.save(small_icon, format="PNG", compress_level=0)
|
||||
|
||||
# Imagen JPG Mediana
|
||||
jpg = Image.new("RGB", (700, 450), (255, 255, 255))
|
||||
draw = ImageDraw.Draw(jpg)
|
||||
for i in range(20):
|
||||
x0 = 20 + i * 12
|
||||
y0 = 40 + i * 8
|
||||
x1 = x0 + 160
|
||||
y1 = y0 + 80
|
||||
draw.rectangle([x0, y0, x1, y1], outline=(30, 90, 180), width=3)
|
||||
draw.text((40, 40), "PHOTO JPG", fill=(0, 0, 0))
|
||||
jpg.save(photo_jpg, format="JPEG", quality=85, optimize=True)
|
||||
|
||||
return {
|
||||
"big_png": big_png,
|
||||
"small_icon": small_icon,
|
||||
"photo_jpg": photo_jpg,
|
||||
}
|
||||
|
||||
@pytest.fixture
|
||||
def base_cfg(tmp_path: Path) -> Dict[str, Dict]:
|
||||
return {
|
||||
"paths": {
|
||||
"input_dir": str(tmp_path / "input"),
|
||||
"output_dir": str(tmp_path / "output"),
|
||||
"temp_dir": str(tmp_path / "temp"),
|
||||
},
|
||||
"filters": {
|
||||
"min_kb": 1,
|
||||
"min_width": 0,
|
||||
"min_height": 0,
|
||||
},
|
||||
"dedup": {
|
||||
"enabled": True,
|
||||
},
|
||||
"output": {
|
||||
"format": "zip",
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"log_file": str(tmp_path / "logs" / "run.log"),
|
||||
"format": "%(asctime)s | %(levelname)s | %(message)s",
|
||||
"datefmt": "%Y-%m-%d %H:%M:%S",
|
||||
},
|
||||
}
|
||||
|
||||
# Documento de prueba DOCX
|
||||
def create_docx(path: Path, image_paths: list[Path]) -> Path:
|
||||
doc = Document()
|
||||
doc.add_heading("TEST DOCX", level=1)
|
||||
|
||||
for image_path in image_paths:
|
||||
doc.add_paragraph(image_path.name)
|
||||
doc.add_picture(str(image_path))
|
||||
doc.save(str(path))
|
||||
return path
|
||||
|
||||
# Documento de prueba PDF
|
||||
def create_pdf(path: Path, image_path: Path) -> Path:
|
||||
doc = fitz.open()
|
||||
page = doc.new_page(width=595, height=842)
|
||||
rect = fitz.Rect(72, 72, 420, 360)
|
||||
page.insert_image(rect, stream=image_path.read_bytes())
|
||||
doc.save(str(path))
|
||||
doc.close()
|
||||
return path
|
||||
|
||||
|
||||
# Documento de prueba PPTX
|
||||
def create_fake_pptx(path: Path, media: dict[str, bytes]) -> Path:
|
||||
with zipfile.ZipFile(path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
|
||||
zf.writestr("[Content_Types].xml", "<Types></Types>")
|
||||
|
||||
for filename, data in media.items():
|
||||
zf.writestr(f"ppt/media/{filename}", data)
|
||||
|
||||
return path
|
||||
|
||||
|
||||
# Documento de prueba XLSX
|
||||
def create_fake_xlsx(path: Path, media: dict[str, bytes]) -> Path:
|
||||
with zipfile.ZipFile(path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
|
||||
zf.writestr("[Content_Types].xml", "<Types></Types>")
|
||||
|
||||
for filename, data in media.items():
|
||||
zf.writestr(f"xl/media/{filename}", data)
|
||||
|
||||
return path
|
||||
|
||||
|
||||
# Creacion XML Base
|
||||
def create_empty_ooxml(path: Path) -> Path:
|
||||
with zipfile.ZipFile(path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
|
||||
zf.writestr("[Content_Types].xml", "<Types></Types>")
|
||||
|
||||
return path
|
||||
|
||||
|
||||
def zip_names(zip_path: Path) -> list[str]:
|
||||
with zipfile.ZipFile(zip_path, "r") as zf:
|
||||
return zf.namelist()
|
||||
@@ -0,0 +1,97 @@
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from .conftest import create_fake_pptx
|
||||
|
||||
def test_cli_help_runs_successfully():
|
||||
repo_root = Path(__file__).resolve().parents[1]
|
||||
|
||||
result = subprocess.run(
|
||||
[sys.executable, "main.py", "--help"],
|
||||
cwd=repo_root,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
assert result.returncode == 0
|
||||
assert "--input" in result.stdout
|
||||
assert "--recursive" in result.stdout
|
||||
assert "--min-kb" in result.stdout
|
||||
|
||||
def test_cli_format_folder(tmp_path: Path, sample_images):
|
||||
repo_root = Path(__file__).resolve().parents[1]
|
||||
|
||||
input_dir = tmp_path / "input"
|
||||
output_dir = tmp_path / "output"
|
||||
temp_dir = tmp_path / "temp"
|
||||
logs_dir = tmp_path / "logs"
|
||||
|
||||
input_dir.mkdir()
|
||||
|
||||
pptx_path = input_dir / "folder_mode.pptx"
|
||||
create_fake_pptx(
|
||||
pptx_path,
|
||||
{
|
||||
# Usamos JPG para asegurar que pase filtros por tamaño
|
||||
"photo.jpg": sample_images["photo_jpg"].read_bytes(),
|
||||
},
|
||||
)
|
||||
|
||||
config_path = tmp_path / "config.json"
|
||||
config_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"paths": {
|
||||
"input_dir": str(input_dir),
|
||||
"output_dir": str(output_dir),
|
||||
"temp_dir": str(temp_dir),
|
||||
},
|
||||
"filters": {
|
||||
"min_kb": 1,
|
||||
"min_width": 0,
|
||||
"min_height": 0,
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"log_file": str(logs_dir / "run.log"),
|
||||
},
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"main.py",
|
||||
"--config",
|
||||
str(config_path),
|
||||
"--input",
|
||||
str(pptx_path),
|
||||
"--output",
|
||||
str(output_dir),
|
||||
"--temp-dir",
|
||||
str(temp_dir),
|
||||
"--format",
|
||||
"folder",
|
||||
"--min-kb",
|
||||
"1",
|
||||
"--log-file",
|
||||
str(logs_dir / "run.log"),
|
||||
],
|
||||
cwd=repo_root,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
|
||||
output_folder = output_dir / "folder_mode"
|
||||
assert output_folder.exists()
|
||||
assert output_folder.is_dir()
|
||||
|
||||
extracted_files = list(output_folder.iterdir())
|
||||
assert len(extracted_files) == 1
|
||||
|
||||
assert (logs_dir / "run.log").exists()
|
||||
@@ -0,0 +1,85 @@
|
||||
from pathlib import Path
|
||||
from document_image_extractor.pipeline import process_file
|
||||
from .conftest import (create_docx, create_pdf, create_fake_pptx, create_fake_xlsx, zip_names)
|
||||
|
||||
def test_docx_extracts_real_extensions(tmp_path: Path, sample_images, base_cfg):
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
|
||||
docx_path = input_dir / "mixed.docx"
|
||||
create_docx(docx_path, [sample_images["photo_jpg"], sample_images["big_png"]])
|
||||
|
||||
stats = process_file(docx_path, base_cfg)
|
||||
|
||||
assert stats["saved"] == 2
|
||||
|
||||
output_zip = Path(base_cfg["paths"]["output_dir"]) / "mixed.zip"
|
||||
assert output_zip.exists()
|
||||
|
||||
names = zip_names(output_zip)
|
||||
|
||||
assert any(name.endswith(".jpg") for name in names)
|
||||
assert any(name.endswith(".png") for name in names)
|
||||
|
||||
def test_pdf_extracts_image(tmp_path: Path, sample_images, base_cfg):
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
|
||||
pdf_path = input_dir / "sample.pdf"
|
||||
create_pdf(pdf_path, sample_images["big_png"])
|
||||
|
||||
stats = process_file(pdf_path, base_cfg)
|
||||
|
||||
assert stats["found"] >= 1
|
||||
assert stats["saved"] >= 1
|
||||
|
||||
output_zip = Path(base_cfg["paths"]["output_dir"]) / "sample.zip"
|
||||
assert output_zip.exists()
|
||||
|
||||
def test_pptx_extracts_media_images(tmp_path: Path, sample_images, base_cfg):
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
|
||||
pptx_path = input_dir / "slides.pptx"
|
||||
|
||||
create_fake_pptx(
|
||||
pptx_path,
|
||||
{
|
||||
"image1.png": sample_images["big_png"].read_bytes(),
|
||||
"image2.jpg": sample_images["photo_jpg"].read_bytes(),
|
||||
},
|
||||
)
|
||||
|
||||
stats = process_file(pptx_path, base_cfg)
|
||||
|
||||
assert stats["found"] == 2
|
||||
assert stats["saved"] == 2
|
||||
|
||||
output_zip = Path(base_cfg["paths"]["output_dir"]) / "slides.zip"
|
||||
assert output_zip.exists()
|
||||
|
||||
names = zip_names(output_zip)
|
||||
assert any(name.endswith(".png") for name in names)
|
||||
assert any(name.endswith(".jpg") for name in names)
|
||||
|
||||
def test_xlsx_extracts_media_images(tmp_path: Path, sample_images, base_cfg):
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
|
||||
xlsx_path = input_dir / "book.xlsx"
|
||||
|
||||
create_fake_xlsx(
|
||||
xlsx_path,
|
||||
{
|
||||
"image1.png": sample_images["big_png"].read_bytes(),
|
||||
"image2.jpg": sample_images["photo_jpg"].read_bytes(),
|
||||
},
|
||||
)
|
||||
|
||||
stats = process_file(xlsx_path, base_cfg)
|
||||
|
||||
assert stats["found"] == 2
|
||||
assert stats["saved"] == 2
|
||||
|
||||
output_zip = Path(base_cfg["paths"]["output_dir"]) / "book.zip"
|
||||
assert output_zip.exists()
|
||||
@@ -0,0 +1,137 @@
|
||||
from pathlib import Path
|
||||
from document_image_extractor.pipeline import list_input_files, process_file
|
||||
from .conftest import create_empty_ooxml, create_fake_pptx, zip_names
|
||||
|
||||
def test_dedup_removes_duplicate_media(tmp_path: Path, sample_images, base_cfg):
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
|
||||
pptx_path = input_dir / "duplicates.pptx"
|
||||
duplicate_data = sample_images["big_png"].read_bytes()
|
||||
|
||||
create_fake_pptx(
|
||||
pptx_path,
|
||||
{
|
||||
"image1.png": duplicate_data,
|
||||
"image2.png": duplicate_data,
|
||||
},
|
||||
)
|
||||
|
||||
stats = process_file(pptx_path, base_cfg)
|
||||
|
||||
assert stats["found"] == 2
|
||||
assert stats["saved"] == 1
|
||||
assert stats["duplicates"] == 1
|
||||
|
||||
output_zip = Path(base_cfg["paths"]["output_dir"]) / "duplicates.zip"
|
||||
assert output_zip.exists()
|
||||
|
||||
assert len(zip_names(output_zip)) == 1
|
||||
|
||||
|
||||
def test_dimension_filter_removes_small_icons(tmp_path: Path, sample_images, base_cfg):
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
|
||||
base_cfg["filters"]["min_kb"] = 1
|
||||
base_cfg["filters"]["min_width"] = 100
|
||||
base_cfg["filters"]["min_height"] = 100
|
||||
|
||||
pptx_path = input_dir / "filters.pptx"
|
||||
|
||||
create_fake_pptx(
|
||||
pptx_path,
|
||||
{
|
||||
"big.png": sample_images["big_png"].read_bytes(),
|
||||
"icon.png": sample_images["small_icon"].read_bytes(),
|
||||
},
|
||||
)
|
||||
|
||||
stats = process_file(pptx_path, base_cfg)
|
||||
|
||||
assert stats["found"] == 2
|
||||
assert stats["saved"] == 1
|
||||
assert stats["filtered_dims"] == 1
|
||||
|
||||
output_zip = Path(base_cfg["paths"]["output_dir"]) / "filters.zip"
|
||||
assert output_zip.exists()
|
||||
|
||||
names = zip_names(output_zip)
|
||||
assert len(names) == 1
|
||||
|
||||
|
||||
def test_empty_ooxml_does_not_create_zip(tmp_path: Path, base_cfg):
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
|
||||
pptx_path = input_dir / "empty.pptx"
|
||||
create_empty_ooxml(pptx_path)
|
||||
|
||||
stats = process_file(pptx_path, base_cfg)
|
||||
|
||||
assert stats["found"] == 0
|
||||
assert stats["saved"] == 0
|
||||
|
||||
output_zip = Path(base_cfg["paths"]["output_dir"]) / "empty.zip"
|
||||
assert not output_zip.exists()
|
||||
|
||||
|
||||
def test_corrupt_file_returns_error_and_cleans_temp(tmp_path: Path, base_cfg):
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
|
||||
corrupt_path = input_dir / "corrupt.xlsx"
|
||||
corrupt_path.write_bytes(b"this is not a valid zip file")
|
||||
|
||||
stats = process_file(corrupt_path, base_cfg)
|
||||
|
||||
assert "error" in stats
|
||||
|
||||
temp_folder = Path(base_cfg["paths"]["temp_dir"]) / "corrupt"
|
||||
assert not temp_folder.exists()
|
||||
|
||||
|
||||
def test_list_input_files_recursive(tmp_path: Path, sample_images):
|
||||
input_dir = tmp_path / "input"
|
||||
subfolder = input_dir / "subfolder"
|
||||
subfolder.mkdir(parents=True)
|
||||
|
||||
pptx_path = subfolder / "nested.pptx"
|
||||
create_fake_pptx(
|
||||
pptx_path,
|
||||
{
|
||||
"image1.png": sample_images["big_png"].read_bytes(),
|
||||
},
|
||||
)
|
||||
|
||||
non_recursive = list_input_files(input_dir, recursive=False)
|
||||
recursive = list_input_files(input_dir, recursive=True)
|
||||
|
||||
assert non_recursive == []
|
||||
assert recursive == [pptx_path]
|
||||
|
||||
|
||||
def test_format_folder_creates_folder_output(tmp_path: Path, sample_images, base_cfg):
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
|
||||
base_cfg["output"]["format"] = "folder"
|
||||
|
||||
pptx_path = input_dir / "folder_output.pptx"
|
||||
create_fake_pptx(
|
||||
pptx_path,
|
||||
{
|
||||
"image1.png": sample_images["big_png"].read_bytes(),
|
||||
},
|
||||
)
|
||||
|
||||
stats = process_file(pptx_path, base_cfg)
|
||||
|
||||
assert stats["saved"] == 1
|
||||
|
||||
output_folder = Path(base_cfg["paths"]["output_dir"]) / "folder_output"
|
||||
assert output_folder.exists()
|
||||
assert output_folder.is_dir()
|
||||
|
||||
extracted_files = list(output_folder.iterdir())
|
||||
assert len(extracted_files) == 1
|
||||
Reference in New Issue
Block a user