xiaoxue_admin
/
medgraph-engine
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

同步完整源码 - 2026-05-25

This commit is contained in:
xiaoxue 2026-05-25 00:32:35 +08:00
commit f4e903630a
51 changed files with 8271 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
.env.example Normal file
View File

@ -0,0 +1,24 @@
# Neo4j — Choose one:
# Option A: Docker local (run docker-compose up -d)
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=changeme-local-password
NEO4J_DATABASE=neo4j
# Option B: AuraDB Free (create at https://neo4j.com/cloud/aura-free/)
# NEO4J_URI=neo4j+s://your-instance.databases.neo4j.io
# NEO4J_USERNAME=neo4j
# NEO4J_PASSWORD=your-auradb-password
# NEO4J_DATABASE=neo4j
# Google AI Studio — Get your free key at https://aistudio.google.com/apikey
GCP_API_KEY=your-google-ai-studio-key
# MedGraph API
API_KEY=your-medgraph-api-key
ENVIRONMENT=development
# Optional: Only needed for entity extraction and intelligent router
# Requires a Google Cloud Platform project with Vertex AI enabled
# GCP_PROJECT=your-gcp-project

28
.gitignore vendored Normal file
View File

@ -0,0 +1,28 @@
# Environment
.env
# Python
__pycache__/
*.pyc
*.pyo
# Data (copyrighted book content — not open source)
parsed/
extracted/
data/processed/
data/enriched/
# Runtime
dedup_checkpoint.json
node_modules/
# IDE
.vscode/
.idea/
# OS
.DS_Store
Thumbs.db
# TP Informatica (not part of MedGraph)
TP INFORMATICA/

42
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,42 @@
# Contributing to MedGraph
Thanks for your interest in contributing! Here's how you can help.
## Ways to Contribute
- **Report bugs** — Open an issue describing the problem, steps to reproduce, and expected behavior
- **Suggest features** — Open an issue with your idea and how it would improve the system
- **Create DAGs** — Clinical reasoning flows for new medical topics (YAML format in `dags/`)
- **Improve the parser** — Better structure detection for different book formats
- **Improve entity extraction** — Better prompts, validation, or canonicalization
- **Add ontology mappings** — Extend ATC/SNOMED coverage
- **Documentation** — Fix typos, improve explanations, add examples
## Development Setup
1. Fork the repository
2. Clone your fork: `git clone https://github.com/your-username/medgraph.git`
3. Install dependencies: `pip install -r requirements.txt`
4. Copy `.env.example` to `.env` and configure your credentials
5. Create a branch: `git checkout -b feature/my-feature`
6. Make your changes
7. Test locally
8. Commit and push
9. Open a Pull Request
## Code Style
- Python code follows standard conventions
- No hardcoded credentials — use environment variables
- Functions should have docstrings explaining what they do
- Keep it simple — avoid over-engineering
## Data and Copyright
- **Do NOT commit** copyrighted book content (PDFs, parsed chunks, extracted entities)
- **Do NOT commit** credentials or API keys
- DAGs, schema definitions, and pipeline code are fine to commit
## Questions?
Open an issue on GitHub.

235
LICENSE Normal file
View File

@ -0,0 +1,235 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users.
When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software.
A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public.
The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version.
An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license.
The precise terms and conditions for copying, distribution and modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based on the Program.
To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work.
A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source.
The Corresponding Source for a work in source code form is that same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures.
When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified it, and giving a relevant date.
b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices".
c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so.
A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways:
a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b.
d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d.
A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product.
"Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made.
If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM).
The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or authors of the material; or
e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors.
All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11).
However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.
Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party.
If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it.
A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph.
Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation.
If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program.
Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source. For example, if your program is a web application, its interface could display a "Source" link that leads users to an archive of the code. There are many ways you could offer source, and different solutions will be better for different programs; see section 13 for the specific requirements.
You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU AGPL, see <http://www.gnu.org/licenses/>.

488
README.md Normal file
View File

@ -0,0 +1,488 @@
# MedGraph
**A knowledge graph engine that transforms textbooks and documents into an intelligent, queryable system with semantic search, reasoning flows, and ontological classification.**
Originally built for medical education, but the pipeline is domain-agnostic — it works with any field where knowledge is structured in books: law, engineering, biology, chemistry, or any academic discipline. Bring your own books, build your own knowledge graph.
---
## The Problem
Students and professionals deal with thousands of pages across dozens of textbooks. Information is fragmented across multiple sources with no way to ask cross-cutting questions and get a unified answer backed by exact page citations from multiple books.
## The Solution
MedGraph parses PDFs (textbooks, papers, course materials), chunks them semantically, generates vector embeddings, extracts entities and relationships using LLMs, maps them to standard ontologies, and exposes everything through an intelligent API that a language model can query.
It's not a search engine. It's a **knowledge graph with structured reasoning**.
> While the examples and built-in ontologies (ATC/SNOMED) are medical, the core pipeline — parsing, chunking, embedding, entity extraction, and graph construction — works with any domain.
---
## Requirements
| Requirement | What it is | Cost | Where to get it |
|---|---|---|---|
| **Python 3.10+** | Runtime | Free | [python.org](https://python.org) |
| **Docker** (optional) | For running Neo4j locally | Free | [docker.com](https://docker.com) |
| **Neo4j** | Graph database | Free | Docker (local) or [AuraDB Free](https://neo4j.com/cloud/aura-free/) |
| **Google AI Studio API key** | For embeddings and entity extraction | Free | [aistudio.google.com/apikey](https://aistudio.google.com/apikey) |
| **Your own PDFs** | The content you want to process | — | Your textbooks, papers, course materials |
No credit card required. Everything runs locally.
### Neo4j: Local vs Cloud
| | Docker (local) | AuraDB Free (cloud) |
|---|---|---|
| Cost | Free | Free |
| Limits | None | 200K nodes, 400K relationships |
| Access | Your PC only | From anywhere |
| Setup | `docker-compose up -d` | Create account at neo4j.io |
| URI | `bolt://localhost:7687` | `neo4j+s://xxx.databases.neo4j.io` |
| Best for | Getting started, development | Sharing, production |
### Google AI Studio API Key
The pipeline uses Google's Gemini models for embeddings, entity extraction, and query routing. All you need is a free API key:
1. Go to [aistudio.google.com/apikey](https://aistudio.google.com/apikey)
2. Click "Create API Key"
3. Copy the key to your `.env` file
No GCP project, no credit card, no billing account required. The free tier is enough to process several books.
---
## Quick Start (5 minutes)
### Step 1 — Clone and configure
```bash
git clone https://github.com/robincanito/medgraph-engine.git
cd medgraph-engine
pip install -r requirements.txt
cp .env.example .env
```
Edit `.env` with your credentials:
```
# Get your free API key at https://aistudio.google.com/apikey
GCP_API_KEY=your-google-ai-studio-key
# If using Docker (local Neo4j):
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=changeme-local-password
# If using AuraDB Free (cloud Neo4j):
# NEO4J_URI=neo4j+s://your-instance.databases.neo4j.io
# NEO4J_USERNAME=neo4j
# NEO4J_PASSWORD=your-auradb-password
```
### Step 2 — Start Neo4j
**Option A — Docker (recommended):**
```bash
docker-compose up -d
# Wait ~30 seconds for Neo4j to be healthy
# Web UI available at http://localhost:7474
```
**Option B — AuraDB Free:**
1. Create a free instance at [neo4j.com/cloud/aura-free](https://neo4j.com/cloud/aura-free/)
2. Copy the connection URI and password to your `.env`
### Step 3 — Run the pipeline
```bash
# Place a PDF in the examples/ folder
python quickstart.py
```
`quickstart.py` runs the full pipeline automatically:
1. Checks your environment and dependencies
2. Creates Neo4j schema and indexes
3. Parses your PDF (text extraction, structure detection)
4. Generates 280-word chunks with overlap
5. Creates vector embeddings (Gemini 3072d)
6. Extracts entities with LLM (requires Vertex AI — skipped if not configured)
7. Verifies everything works
### Step 4 — Start the API
```bash
cd api && uvicorn main:app --reload
# Open http://localhost:8000/docs
```
### Step 5 — Connect your LLM
See the [MedGraph Client](https://github.com/robincanito/medgraph-client-oss) for connecting Claude Code, ChatGPT, Ollama, or OpenClaw to your instance.
---
## Deploy to Cloud (optional)
If you want your API accessible from anywhere (other devices, bots, OpenClaw via WhatsApp), you can deploy to Google Cloud Run:
### Prerequisites
- [Google Cloud CLI](https://cloud.google.com/sdk/docs/install) installed
- A GCP project (free tier works)
- Neo4j on AuraDB Free (Docker local won't work with Cloud Run — it needs a remote database)
### Steps
```bash
# 1. Login to Google Cloud
gcloud auth login
# 2. Deploy the API
cd api
gcloud run deploy medgraph-api \
--source . \
--project your-gcp-project \
--region us-central1 \
--set-env-vars="NEO4J_URI=neo4j+s://your-auradb.databases.neo4j.io,NEO4J_USERNAME=neo4j,NEO4J_PASSWORD=your-password,NEO4J_DATABASE=neo4j,API_KEY=your-api-key,GCP_API_KEY=your-ai-studio-key,ENVIRONMENT=production"
# 3. Done — Cloud Run gives you a public URL
# Example: https://medgraph-api-xxxxx.us-central1.run.app
```
Your API is now live. Point the [MedGraph Client](https://github.com/robincanito/medgraph-client-oss) to that URL and query from anywhere.
> **Tip:** For production, use [Google Secret Manager](https://cloud.google.com/secret-manager) instead of passing credentials as env vars. Replace `--set-env-vars` with `--set-secrets` for each sensitive value.
---
## Architecture
```
Layer 1: DOCUMENTAL Chunks with vector embeddings (semantic search)
Layer 2: SEMANTIC Full-text BM25 + RRF fusion + query rewriting
Layer 3: GRAPH Typed entities + relationships (CAUSED_BY, TREATED_WITH, etc.)
Layer 4: REASONING DAGs: directed clinical flows (symptom -> diagnosis -> treatment)
Layer 5: ONTOLOGY ATC drug classification + SNOMED clinical terminology
```
Each layer enriches the previous one. An intelligent router (Gemini) analyzes each query and activates the relevant layers automatically.
### System Diagram
```
User Question
|
v
GEMINI ROUTER (analyzer)
|
+---> ONTOLOGY -----> ATC hierarchy / SNOMED classification
|
+---> GRAPH --------> Entity relationships (CAUSED_BY, TREATED_WITH, etc.)
|
+---> BIBLIOGRAPHY -> Hybrid search: vector (3072d) + BM25 + RRF fusion
|
+---> ACTIVITIES ---> Academic activities (labs, seminars, practical work)
|
+---> DAGS ---------> Clinical reasoning flows (pathways + clinical decisions)
|
v
UNIFIED RESPONSE (structured JSON with sources)
```
---
## What You Can Build
The engine scales with the content you feed it. As a reference, a deployment with 17 textbooks produces:
| Metric | Example at scale |
|---|---|
| Semantic entities | 100K+ (pathologies, drugs, anatomy, signs, procedures) |
| Typed relationships | 1M+ (CAUSED_BY, TREATED_WITH, DIAGNOSED_WITH, etc.) |
| Text chunks | ~2,000 per book (280 words each, with overlap) |
| Ontology mappings | ATC drug hierarchy + SNOMED clinical terminology |
| Clinical DAGs | Custom reasoning flows per topic |
| Embedding dimensions | 3,072 (Gemini Embedding 2 Preview) |
Your instance starts empty. Each book you process through the pipeline adds thousands of entities and relationships automatically.
---
## Tech Stack
| Component | Technology |
|---|---|
| Knowledge Graph | Neo4j AuraDB |
| Embeddings | Google Gemini Embedding 2 Preview (3072 dims) |
| Entity Extraction | Gemini 3.1 Flash Lite |
| Query Router | Gemini 2.5 Flash |
| API | FastAPI on Google Cloud Run |
| Vector Search | Neo4j native vector index |
| Full-text Search | Neo4j native full-text index (BM25) |
| PDF Parsing | PyMuPDF + Google Cloud Vision (OCR) |
| Auth | Bearer token + Google Secret Manager |
| LLM Integration | MCP Server (Claude) + Custom GPT (ChatGPT) |
---
## Pipeline
The complete ingestion pipeline transforms a PDF into queryable knowledge:
```
PDF (textbook or course material)
|
v
1. PARSE (parser_v2.py)
PyMuPDF extracts text, detects chapters/sections
Cloud Vision API for scanned books (OCR)
|
v
2. CHUNK (parser_v2.py)
280 words target, 60 words overlap
Parent-child structure (parent = full section, children = chunks)
Rich metadata: book, chapter, section, page range, word count
|
v
3. UPLOAD (upload_chunks.py)
Batch upload to Neo4j as :Chunk and :ParentChunk nodes
|
v
4. VECTORIZE (vectorize.py)
Generate embeddings with Gemini Embedding 2 Preview (3072 dims)
Store as node property in Neo4j
|
v
5. EXTRACT ENTITIES (extract_entities.py)
LLM reads each chunk and extracts:
- Entities: pathologies, drugs, anatomy, signs, symptoms, procedures...
- Relationships: CAUSED_BY, TREATED_WITH, DIAGNOSED_WITH...
Canonicalize (deduplicate, merge synonyms, sum frequencies)
Upload as typed nodes + MENTIONS relationships
|
v
6. MAP ONTOLOGY (ontology.py)
Map drugs to ATC hierarchy (Anatomical Therapeutic Chemical)
Map pathologies/anatomy/procedures to SNOMED-CT
Create IS_A hierarchical relationships
```
---
## Entity Types
| Label | Count | Examples |
|---|---|---|
| Patologia | 36,021 | Otitis media, hypertension, lymphoma |
| EstructuraAnatomica | 12,850 | Tympanic membrane, cochlea, retina |
| Procedimiento | 10,449 | Otoscopy, ECG, chest X-ray |
| Hallazgo | 8,238 | Bulging TM, ST elevation |
| Farmaco | 7,070 | Amoxicillin, ibuprofen, oseltamivir |
| Signo | 5,945 | Fever, edema, cyanosis |
| Parametro | 5,298 | Blood pressure, heart rate, hemoglobin |
| MetodoDx | 5,222 | CBC, echocardiogram, audiometry |
| Agente | 4,135 | S. pneumoniae, EBV, Influenza A |
| GrupoFarmacologico | 3,993 | NSAIDs, beta-lactams, coxibs |
| Sintoma | 3,017 | Otalgia, headache, dyspnea |
## Relationship Types
`CAUSED_BY` `TREATED_WITH` `DIAGNOSED_WITH` `MANIFESTS_WITH` `PART_OF` `EVALUATES` `BELONGS_TO` `CAN_PRODUCE` `DIFFERENTIAL_OF` `ASSOCIATED_WITH` `RISK_FACTOR` `COMPLICATION_OF` `VARIANT_OF` `IS_A` `MENTIONS` `PATHWAY` `CLINICAL`
---
## API Endpoints
### Main (use this)
```bash
POST /query
# Intelligent unified query — analyzes the question and activates relevant layers
{"pregunta": "What NSAIDs are contraindicated in chronic kidney disease?", "top_k": 10}
```
### Specific
```bash
POST /search/hybrid # Hybrid search (semantic + BM25 + RRF)
GET /topic/{topic}/comprehensive # Full topic: graph + activities + bibliography
GET /pathology/{name} # Pathology details from graph
GET /procedure/{name} # Procedure details from graph
GET /activity/search/{name} # Academic activity search
GET /topic/{topic}/ontology # ATC/SNOMED hierarchical classification
GET /topic/{topic}/pathways # Clinical reasoning DAGs
GET /topic/{topic}/clinical # Clinical decision flows
```
---
## DAGs: Clinical Reasoning
DAGs (Directed Acyclic Graphs) add **sequence and clinical logic** on top of the knowledge graph. The graph knows that "OMA is connected to otalgia, otoscopy, amoxicillin, S. pneumoniae". The DAG adds the **order**: symptom -> exam -> finding -> diagnosis -> treatment -> follow-up.
Example — Otalgia management:
```
Otalgia -> Anamnesis (age, duration, fever, ENT history)
-> Otoscopy
-> IF swollen CAE, positive tragus sign -> Otitis externa
-> IF bulging, erythematous, opaque TM -> Acute otitis media
-> IF retracted TM, air-fluid level -> Effusive otitis media
-> IF normal TM and CAE -> Referred otalgia
-> Treatment per diagnosis
-> Follow-up 48-72h
-> Escalation if no improvement
```
---
## MCP Server (Claude Integration)
MedGraph includes an MCP server that connects directly to Claude (Anthropic's AI), giving it real-time access to the medical knowledge graph.
```bash
# Install and run
pip install fastmcp httpx
python mcp_server.py
```
Available tools: `medgraph_query`, `medgraph_search`, `medgraph_comprehensive`, `medgraph_pathology`, `medgraph_procedure`, `medgraph_activity`, `medgraph_pathways`, `medgraph_clinical`, `medgraph_cronograma`
---
## Getting Started
### Prerequisites
- Python 3.10+
- Neo4j AuraDB instance (or local Neo4j)
- Google Cloud Platform account (for Gemini embeddings + Vertex AI)
### Installation
```bash
git clone https://github.com/robincanito/medgraph.git
cd medgraph
pip install -r requirements.txt
cp .env.example .env
# Edit .env with your credentials
```
### Run the pipeline on your own books
```bash
# 1. Parse a PDF
python parser_v2.py my-textbook
# 2. Upload chunks to Neo4j
python upload_chunks.py my-textbook
# 3. Generate embeddings
python vectorize.py my-textbook
# 4. Extract entities with LLM
python extract_entities.py my-textbook
# 5. Map to ATC/SNOMED ontology
python ontology.py
```
### Run the API
```bash
cd api
uvicorn main:app --reload
```
### Deploy to Cloud Run
```bash
cd api
gcloud run deploy medgraph-api --source . --project your-project --region us-central1
```
---
## Important Note
**This repository contains the engine only.** Book data, extracted entities, parsed chunks, and the Neo4j database are not included due to copyright restrictions on medical textbooks. You must provide your own PDFs and run the pipeline to populate your own knowledge graph.
### Disclaimer
MedGraph is a software tool for processing and structuring text content. Users are solely responsible for ensuring they have the appropriate rights to any content they process through the system. The authors of MedGraph do not endorse, facilitate, or assume liability for copyright infringement or any unauthorized use of copyrighted materials.
---
## Security Considerations
MedGraph is designed for local development and personal use out of the box. **If you plan to deploy it to production or expose the API publicly, review the following:**
- **API authentication:** The API uses a single Bearer token (`API_KEY` env var). For production, implement proper auth (OAuth2, JWT, or API key rotation).
- **CORS:** The default config allows `localhost` origins only. Add your frontend domains explicitly — never use `allow_origins=["*"]` in production.
- **Rate limiting:** Included via `slowapi` (100 req/min default). Adjust for your expected load.
- **Neo4j credentials:** Use environment variables or a secret manager (e.g., Google Secret Manager, AWS Secrets Manager). Never hardcode credentials.
- **HTTPS:** Cloud Run and most hosting providers handle TLS automatically. If self-hosting, put the API behind a reverse proxy (nginx/Caddy) with SSL.
- **Input validation:** The API accepts user queries as strings. While they go to an LLM (not SQL), consider sanitizing inputs if exposing publicly.
- **Docker Compose:** The default Neo4j password is `changeme-local-password`. Change it immediately if exposing the database.
- **Dependency updates:** Run `pip audit` periodically to check for known vulnerabilities in dependencies.
This is not an exhaustive security review. For production deployments, conduct a proper security assessment based on your specific infrastructure and threat model.
## Tests
```bash
pip install pytest
pytest tests/ -v
```
---
## Project Structure
```
medgraph/
api/ # FastAPI application
routers/ # API route handlers
services/ # Business logic (graph, vector, analyzer)
main.py # App entry point
dags/ # Clinical reasoning flows (YAML)
parser_v2.py # PDF parser with structure detection
upload_chunks.py # Neo4j chunk uploader
vectorize.py # Embedding generator
extract_entities.py # LLM entity extractor
ontology.py # ATC/SNOMED mapper
load_dags.py # DAG loader
db.py # Neo4j connection helper
schema.py # Database schema and indexes
mcp_server.py # MCP server for Claude
dedup_entities.py # Entity deduplication
catalog.json # Book metadata catalog
requirements.txt # Python dependencies
```
---
## Origin
Born from the frustration of studying across dozens of fragmented textbooks and the conviction that medical knowledge should be structured, connected, and queryable — not trapped in isolated PDFs.
---
## License
This project is licensed under the **GNU Affero General Public License v3.0** (AGPLv3).
You are free to use, modify, and distribute this software. If you deploy it as a network service, you must make your modified source code available to users of that service.
See [LICENSE](LICENSE) for the full text.
---
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

10
api/Dockerfile Normal file
View File

@ -0,0 +1,10 @@
FROM python:3.13-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

122
api/main.py Normal file
View File

@ -0,0 +1,122 @@
"""MedGraph API — Knowledge base medica para estudio."""
import os
import time
from fastapi import FastAPI, Request, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.util import get_remote_address
from slowapi.errors import RateLimitExceeded
from dotenv import load_dotenv
from routers import topics, search, pathology, procedure, activity, admin, comprehensive, ontology_router, unified
from services import graph
load_dotenv()
API_KEY = os.getenv("API_KEY", "")
# Rate limiter
limiter = Limiter(key_func=get_remote_address, default_limits=["100/minute"])
app = FastAPI(
title="MedGraph API",
description="Knowledge base medica con grafo de conocimiento y busqueda semantica",
version="2.0.0",
servers=[{"url": os.getenv("API_BASE_URL", "http://localhost:8000")}],
docs_url="/docs" if os.getenv("ENVIRONMENT") == "development" else None,
redoc_url=None,
)
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
# CORS restrictivo
app.add_middleware(
CORSMiddleware,
allow_origins=[
os.getenv("CORS_ORIGIN_1", "http://localhost:3000"),
# Add your frontend origins here
os.getenv("API_BASE_URL", "http://localhost:8000"),
# Add your frontend origins here
],
allow_methods=["GET", "POST"],
allow_headers=["Authorization", "Content-Type", "X-API-Key"],
)
# === AUTH ===
@app.middleware("http")
async def auth_middleware(request: Request, call_next):
# Skip auth for health and schema
if request.url.path in ("/health", "/chatgpt-schema"):
return await call_next(request)
api_key = (
request.headers.get("X-API-Key")
or request.headers.get("Authorization", "").replace("Bearer ", "")
)
if api_key != API_KEY:
raise HTTPException(status_code=401, detail="Invalid API key")
return await call_next(request)
# === LOGGING ===
@app.middleware("http")
async def log_middleware(request: Request, call_next):
start = time.time()
response = await call_next(request)
duration = time.time() - start
print(f"{request.method} {request.url.path} -> {response.status_code} ({duration:.2f}s)")
return response
# === ROUTERS ===
app.include_router(topics.router)
app.include_router(search.router)
app.include_router(pathology.router)
app.include_router(procedure.router)
app.include_router(activity.router)
app.include_router(admin.router)
app.include_router(comprehensive.router)
app.include_router(ontology_router.router)
app.include_router(unified.router)
# === HEALTH ===
@app.get("/health")
async def health():
try:
graph.query("RETURN 1")
return {"status": "ok", "service": "medgraph-api", "db": "connected"}
except Exception:
from fastapi.responses import JSONResponse
return JSONResponse({"status": "degraded", "service": "medgraph-api", "db": "disconnected"}, 503)
@app.get("/chatgpt-schema")
async def chatgpt_schema():
"""Schema reducido (5 endpoints) para importar en ChatGPT Actions."""
from fastapi.responses import JSONResponse
import json
for path in [
os.path.join(os.path.dirname(os.path.abspath(__file__)), "chatgpt_action_schema.json"),
"/app/chatgpt_action_schema.json",
"chatgpt_action_schema.json",
]:
if os.path.exists(path):
with open(path, "r", encoding="utf-8") as f:
return JSONResponse(content=json.load(f))
return {"error": "schema not found"}
@app.get("/stats")
async def stats():
return graph.get_stats()
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8080)

8
api/requirements.txt Normal file
View File

@ -0,0 +1,8 @@
fastapi>=0.115
uvicorn>=0.34
neo4j>=5.0
google-genai>=1.0
python-dotenv>=1.0
pymupdf>=1.24
python-multipart>=0.0.9
slowapi>=0.1.9

0
api/routers/__init__.py Normal file
View File

49
api/routers/activity.py Normal file
View File

@ -0,0 +1,49 @@
"""Endpoints de actividades: TPs, Seminarios, Talleres, Acreditaciones."""
from fastapi import APIRouter, Query
from services import graph
router = APIRouter(prefix="/activity", tags=["activity"])
@router.get("/search/{nombre}")
async def search_activity(nombre: str):
"""Buscar actividad por nombre, titulo o ID. Ej: 'TP N7', 'Otoscopia', 'Seminario N4'."""
results = graph.get_activity(nombre)
if not results:
return {"error": f"Actividad '{nombre}' no encontrada", "sugerencia": "Probá con 'TP', 'Seminario', 'Taller' + numero o tema"}
return {"results": results, "count": len(results)}
@router.get("/list")
async def list_activities(
up_id: str | None = Query(None, description="Filtrar por UP (ej: COURSE-UNIT2)"),
tipo: str | None = Query(None, description="Filtrar por tipo: tp, seminario, taller, acreditacion, tutoria"),
):
"""Listar todas las actividades, opcionalmente filtradas por UP y/o tipo."""
results = graph.list_activities(up_id=up_id, tipo=tipo)
return {"results": results, "count": len(results)}
@router.get("/{activity_id}/material")
async def get_material(activity_id: str):
"""Obtener el contenido completo de los documentos vinculados a una actividad."""
results = graph.get_activity_material(activity_id)
if not results:
return {"error": f"No hay material vinculado a '{activity_id}'"}
return {"activity_id": activity_id, "documentos": results, "count": len(results)}
@router.get("/doc/{doc_nombre}")
async def get_document(doc_nombre: str):
"""Buscar un documento por nombre. Ej: 'Otoscopia', 'Bioseguridad', 'ECG'."""
results = graph.query("""
MATCH (d:Documento)
WHERE toLower(d.nombre) CONTAINS toLower($nombre)
RETURN d.id AS id, d.nombre AS nombre, d.tipo AS tipo,
d.archivo AS archivo, d.palabras AS palabras, d.texto AS texto
ORDER BY d.nombre
""", {"nombre": doc_nombre})
if not results:
return {"error": f"Documento '{doc_nombre}' no encontrado"}
return {"results": results, "count": len(results)}

43
api/routers/admin.py Normal file
View File

@ -0,0 +1,43 @@
"""Endpoints administrativos: ingesta de libros."""
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks
from services.ingest import run_ingest, get_job
router = APIRouter(prefix="/admin", tags=["admin"])
@router.post("/ingest")
async def ingest_libro(
background_tasks: BackgroundTasks,
libro_id: str = Form(...),
titulo: str = Form(...),
autor: str = Form(""),
pdf: UploadFile = File(...),
):
"""Sube un PDF y arranca el pipeline de ingesta en background."""
existing = get_job(libro_id)
if existing and existing["status"] == "running":
raise HTTPException(409, f"Ya hay una ingesta en curso para {libro_id}")
if not pdf.filename.lower().endswith(".pdf"):
raise HTTPException(400, "Solo se aceptan archivos PDF")
# Save to temp
pdf_path = f"/tmp/{libro_id}.pdf"
content = await pdf.read()
with open(pdf_path, "wb") as f:
f.write(content)
# Run in background
background_tasks.add_task(run_ingest, libro_id, pdf_path, titulo, autor)
return {"job_id": libro_id, "status": "started"}
@router.get("/ingest/{job_id}")
async def ingest_status(job_id: str):
"""Consulta el estado de una ingesta en curso."""
job = get_job(job_id)
if not job:
raise HTTPException(404, "Job no encontrado")
return job

112
api/routers/clinical.py Normal file
View File

@ -0,0 +1,112 @@
"""Endpoints de razonamiento: pathways (fisiopatologia) y clinical (arbol de decision)."""
from fastapi import APIRouter
from services.graph import query
router = APIRouter(tags=["clinical"])
@router.get("/topic/{tema}/pathways")
async def get_pathways(tema: str):
"""Cadenas causales/fisiopatologicas de un tema. Deterministas, lineales."""
# Buscar todos los DAGs PATHWAY que involucren este tema
paths = query("""
MATCH (start)-[r:PATHWAY*1..12]->(end)
WHERE ANY(n IN nodes((start)-[r*1..12]->(end))
WHERE toLower(n.nombre) CONTAINS toLower($tema))
WITH start, r, end,
[rel IN r | {
nombre_dag: rel.nombre_dag,
orden: rel.orden,
nota: rel.nota
}] AS rels,
[n IN nodes((start)-[r*1..12]->(end)) | {
nombre: n.nombre,
tipo: labels(n)[0]
}] AS nodos
RETURN DISTINCT rels[0].nombre_dag AS nombre_dag,
nodos, rels
""", {"tema": tema})
if not paths:
# Fallback: buscar nodos directamente conectados con PATHWAY
paths = query("""
MATCH (a)-[r:PATHWAY]->(b)
WHERE toLower(a.nombre) CONTAINS toLower($tema)
OR toLower(b.nombre) CONTAINS toLower($tema)
RETURN r.nombre_dag AS nombre_dag,
a.nombre AS desde, labels(a)[0] AS tipo_desde,
b.nombre AS hasta, labels(b)[0] AS tipo_hasta,
r.orden AS orden, r.nota AS nota
ORDER BY r.nombre_dag, r.orden
""", {"tema": tema})
# Agrupar por nombre_dag
dags = {}
for p in paths:
dag_name = p["nombre_dag"]
if dag_name not in dags:
dags[dag_name] = []
dags[dag_name].append({
"desde": {"nombre": p["desde"], "tipo": p["tipo_desde"]},
"hasta": {"nombre": p["hasta"], "tipo": p["tipo_hasta"]},
"orden": p["orden"],
"nota": p.get("nota", ""),
})
return {
"tema": tema,
"tipo": "pathways",
"descripcion": "Cadenas causales y fisiopatologicas",
"pathways": [{"nombre": k, "pasos": v} for k, v in dags.items()],
}
return {
"tema": tema,
"tipo": "pathways",
"descripcion": "Cadenas causales y fisiopatologicas",
"pathways": paths,
}
@router.get("/topic/{tema}/clinical")
async def get_clinical(tema: str):
"""Arbol de decision clinica ante un tema. Bifurcaciones con condiciones."""
# Buscar relaciones CLINICAL
steps = query("""
MATCH (a)-[r:CLINICAL]->(b)
WHERE toLower(a.nombre) CONTAINS toLower($tema)
OR toLower(b.nombre) CONTAINS toLower($tema)
OR r.nombre_dag CONTAINS toLower($tema)
RETURN r.nombre_dag AS nombre_dag,
a.nombre AS desde, labels(a)[0] AS tipo_desde,
b.nombre AS hasta, labels(b)[0] AS tipo_hasta,
r.orden AS orden, r.tipo_paso AS tipo_paso,
r.condicion AS condicion, r.nota AS nota
ORDER BY r.nombre_dag, r.orden
""", {"tema": tema})
# Agrupar por nombre_dag
dags = {}
for s in steps:
dag_name = s["nombre_dag"]
if dag_name not in dags:
dags[dag_name] = {"nombre": dag_name, "pasos": []}
dags[dag_name]["pasos"].append({
"desde": {"nombre": s["desde"], "tipo": s["tipo_desde"]},
"hasta": {"nombre": s["hasta"], "tipo": s["tipo_hasta"]},
"orden": s["orden"],
"tipo_paso": s.get("tipo_paso", ""),
"condicion": s.get("condicion", ""),
"nota": s.get("nota", ""),
})
return {
"tema": tema,
"tipo": "clinical",
"descripcion": "Arbol de decision clinica",
"arboles": list(dags.values()),
}

277
api/routers/comprehensive.py Normal file
View File

@ -0,0 +1,277 @@
"""Endpoint comprehensive: todo sobre un tema en un solo call."""
from fastapi import APIRouter, Query
from services import graph, vector
router = APIRouter(tags=["comprehensive"])
@router.get("/topic/{tema}/comprehensive")
async def topic_comprehensive(
tema: str,
top_k: int = Query(default=20, description="Chunks de bibliografia a devolver"),
):
"""Devuelve TODO sobre un tema: grafo + actividades + material + bibliografia.
Orquesta multiples consultas internas y devuelve un paquete completo
para que el LLM no necesite hacer multiples calls.
"""
result = {
"tema": tema,
"grafo": {},
"actividades": [],
"material": [],
"bibliografia": [],
}
# 1. Buscar en el grafo semantico (nodos + relaciones)
# Buscar como Tema
topic_detail = graph.get_topic_detail(tema)
if topic_detail:
result["grafo"]["tema"] = topic_detail
# Buscar como Patologia
patologias = graph.get_pathology(tema)
if patologias:
result["grafo"]["patologias"] = patologias
# Buscar como Procedimiento
procedimientos = graph.get_procedure(tema)
if procedimientos:
result["grafo"]["procedimientos"] = procedimientos
# Buscar entidades Dev (extraidas por LLM)
entidades_dev = _search_dev_entities(tema)
if entidades_dev:
result["grafo"]["entidades_extraidas"] = entidades_dev
# 2. Buscar actividades relacionadas (TPs, seminarios, talleres)
# Buscar con el tema original + sinónimos comunes
search_terms = [tema]
# Agregar sinónimos de entidades encontradas
for ent in entidades_dev:
if ent.get("sinonimos"):
search_terms.extend(ent["sinonimos"][:3])
actividades = []
seen_ids = set()
for term in search_terms:
results_act = graph.get_activity(term)
for a in results_act:
if a.get("id") and a["id"] not in seen_ids:
seen_ids.add(a["id"])
actividades.append(a)
if actividades:
result["actividades"] = actividades
# Para cada actividad, traer su material
for act in actividades:
if act.get("id"):
material = graph.get_activity_material(act["id"])
if material:
for doc in material:
doc["actividad"] = act["nombre"]
result["material"].extend(material)
# 3. Buscar chunks de bibliografia (hybrid search)
try:
search_result = vector.search_hybrid(tema, top_k=top_k)
if search_result.get("results"):
result["bibliografia"] = [
{
"libro": r.get("libro"),
"paginas": f"{r.get('pag_inicio', '?')}-{r.get('pag_fin', '?')}",
"capitulo": r.get("capitulo", ""),
"seccion": r.get("seccion", ""),
"tipo": r.get("tipo", ""),
"texto": r.get("texto", ""),
"score": r.get("rrf_score", 0),
}
for r in search_result["results"]
]
result["busqueda"] = {
"intencion": search_result.get("intencion", ""),
"query_expandida": search_result.get("query_expandida", ""),
}
except Exception:
pass
# 4. Ontología (ATC + SNOMED)
ontologia = {"farmacos_atc": [], "patologias_snomed": [], "query_cruzada": []}
try:
# Fármacos → ATC
farmacos_atc = graph.query("""
MATCH (f:Farmaco)-[:ES_UN*1..5]->(cat:CategoriaATC)
WHERE toLower(f.nombre) CONTAINS toLower($tema)
WITH f.nombre AS farmaco, collect(DISTINCT {codigo: cat.codigo, nombre: cat.nombre, nivel: cat.nivel}) AS jerarquia
RETURN farmaco, jerarquia
ORDER BY farmaco LIMIT 10
""", {"tema": tema})
for r in farmacos_atc:
ontologia["farmacos_atc"].append({
"nombre": r["farmaco"],
"jerarquia": sorted(r["jerarquia"], key=lambda x: x.get("nivel", 0))
})
# Patologías → SNOMED
pato_snomed = graph.query("""
MATCH (p:Patologia)-[:ES_UN*1..3]->(cat:CategoriaSNOMED)
WHERE toLower(p.nombre) CONTAINS toLower($tema)
WITH p.nombre AS patologia, collect(DISTINCT {nombre: cat.nombre, sistema: cat.sistema, nivel: cat.nivel}) AS jerarquia
RETURN patologia, jerarquia
ORDER BY patologia LIMIT 10
""", {"tema": tema})
for r in pato_snomed:
ontologia["patologias_snomed"].append({
"nombre": r["patologia"],
"jerarquia": sorted(r["jerarquia"], key=lambda x: x.get("nivel", 0))
})
# Query cruzada
if ontologia["patologias_snomed"]:
sistemas = set()
for p in ontologia["patologias_snomed"]:
for j in p["jerarquia"]:
if j.get("sistema"):
sistemas.add(j["sistema"])
for sistema in list(sistemas)[:2]:
cruzada = graph.query("""
MATCH (p:Patologia)-[:ES_UN*1..3]->(sno:CategoriaSNOMED {sistema: $sistema})
WHERE toLower(p.nombre) CONTAINS toLower($tema)
MATCH (p)-[:SE_TRATA_CON]->(f:Farmaco)
OPTIONAL MATCH (f)-[:ES_UN*1..5]->(atc:CategoriaATC)
RETURN DISTINCT f.nombre AS farmaco, p.nombre AS patologia,
collect(DISTINCT atc.nombre)[0] AS clase_atc, $sistema AS sistema
LIMIT 10
""", {"tema": tema, "sistema": sistema})
for r in cruzada:
ontologia["query_cruzada"].append({
"farmaco": r["farmaco"],
"patologia": r["patologia"],
"clase_atc": r.get("clase_atc"),
"sistema_snomed": r["sistema"]
})
except Exception:
pass
result["ontologia"] = ontologia
# 5. Resumen de fuentes encontradas
fuentes = set()
for bib in result["bibliografia"]:
if bib.get("libro"):
fuentes.add(bib["libro"])
result["fuentes"] = list(fuentes)
# 6. Flag de completitud
result["tiene_grafo"] = bool(result["grafo"])
result["tiene_actividades"] = bool(result["actividades"])
result["tiene_material"] = bool(result["material"])
result["tiene_bibliografia"] = bool(result["bibliografia"])
result["tiene_ontologia"] = bool(ontologia["farmacos_atc"] or ontologia["patologias_snomed"])
return result
@router.get("/topic/{tema}/pathways")
async def topic_pathways(tema: str):
"""Cadena causal/fisiopatologica: agente -> mecanismo -> efecto -> signo -> complicacion."""
pathways = graph.query("""
MATCH (a)-[r:PATHWAY]->(b)
WHERE toLower(a.nombre) CONTAINS toLower($tema)
OR toLower(b.nombre) CONTAINS toLower($tema)
OR r.nombre_dag CONTAINS toLower($tema)
RETURN DISTINCT r.nombre_dag AS dag, a.nombre AS desde, labels(a)[0] AS tipo_desde,
b.nombre AS hasta, labels(b)[0] AS tipo_hasta,
r.orden AS orden, r.tipo AS tipo_paso, r.nota AS nota
ORDER BY r.nombre_dag, r.orden
""", {"tema": tema})
dags = {}
for p in pathways:
dag_name = p["dag"]
if dag_name not in dags:
dags[dag_name] = {"nombre": dag_name, "pasos": []}
dags[dag_name]["pasos"].append({
"orden": p["orden"],
"desde": p["desde"],
"tipo_desde": p["tipo_desde"],
"hasta": p["hasta"],
"tipo_hasta": p["tipo_hasta"],
"tipo_paso": p.get("tipo_paso"),
"nota": p.get("nota"),
})
return {"tema": tema, "pathways": list(dags.values()), "count": len(dags)}
@router.get("/topic/{tema}/clinical")
async def topic_clinical(tema: str):
"""Arbol de decision clinica con bifurcaciones y condiciones."""
steps = graph.query("""
MATCH (a)-[r:CLINICAL]->(b)
WHERE toLower(a.nombre) CONTAINS toLower($tema)
OR toLower(b.nombre) CONTAINS toLower($tema)
OR r.nombre_dag CONTAINS toLower($tema)
RETURN DISTINCT r.nombre_dag AS dag, a.nombre AS desde, labels(a)[0] AS tipo_desde,
b.nombre AS hasta, labels(b)[0] AS tipo_hasta,
r.orden AS orden, r.tipo_paso AS tipo_paso,
r.condicion AS condicion, r.nota AS nota
ORDER BY r.nombre_dag, r.orden, r.condicion
""", {"tema": tema})
dags = {}
for s in steps:
dag_name = s["dag"]
if dag_name not in dags:
dags[dag_name] = {"nombre": dag_name, "pasos": []}
dags[dag_name]["pasos"].append({
"orden": s["orden"],
"desde": s["desde"],
"tipo_desde": s["tipo_desde"],
"hasta": s["hasta"],
"tipo_hasta": s["tipo_hasta"],
"tipo_paso": s.get("tipo_paso"),
"condicion": s.get("condicion"),
"nota": s.get("nota"),
})
return {"tema": tema, "clinical": list(dags.values()), "count": len(dags)}
def _search_dev_entities(tema: str) -> list:
"""Busca entidades extraidas (Dev) relacionadas con el tema."""
# Buscar en todos los labels Dev
dev_labels = [
"Patologia", "EstructuraAnatomica", "Procedimiento",
"Farmaco", "Agente", "Signo", "Sintoma",
"MetodoDx", "Hallazgo", "GrupoFarmacologico",
"PatologiaDev", "EstructuraAnatomicaDev", "ProcedimientoDev",
"FarmacoDev", "AgenteDev", "SignoDev", "SintomaDev",
"MetodoDxDev", "HallazgoDev", "GrupoFarmacologicoDev",
]
results = []
for label in dev_labels:
try:
matches = graph.query(f"""
MATCH (e:{label})
WHERE toLower(e.nombre) CONTAINS toLower($tema)
OPTIONAL MATCH (e)-[r]-(related)
WHERE NOT related:Chunk
RETURN e.nombre AS nombre, labels(e)[0] AS tipo,
e.sinonimos AS sinonimos, e.freq AS freq,
collect(DISTINCT {{
nombre: related.nombre,
tipo: labels(related)[0],
relacion: type(r)
}})[0..10] AS relaciones
LIMIT 5
""", {"tema": tema})
results.extend(matches)
except Exception:
pass
return results

120
api/routers/ontology_router.py Normal file
View File

@ -0,0 +1,120 @@
"""Endpoint de ontología — traversal jerárquico ATC + SNOMED."""
from fastapi import APIRouter
from services.graph import query as read
router = APIRouter(tags=["ontology"])
@router.get("/topic/{tema}/ontology")
async def get_ontology(tema: str):
"""Clasificación ontológica: jerarquía ATC de fármacos, SNOMED de patologías, queries cruzadas."""
resultado = {
"tema": tema,
"farmacos_atc": [],
"patologias_snomed": [],
"anatomia_snomed": [],
"procedimientos_snomed": [],
"query_cruzada": [],
}
# 1. Fármacos → ATC hierarchy
farmacos = read("""
MATCH (f:Farmaco)-[:ES_UN*1..5]->(cat:CategoriaATC)
WHERE toLower(f.nombre) CONTAINS toLower($tema)
WITH f.nombre AS farmaco, collect(DISTINCT {codigo: cat.codigo, nombre: cat.nombre, nivel: cat.nivel}) AS jerarquia
RETURN farmaco, jerarquia
ORDER BY farmaco
LIMIT 20
""", {"tema": tema})
for r in farmacos:
jerarquia_sorted = sorted(r["jerarquia"], key=lambda x: x.get("nivel", 0))
resultado["farmacos_atc"].append({
"nombre": r["farmaco"],
"jerarquia": jerarquia_sorted
})
# 2. Patologías → SNOMED hierarchy
patologias = read("""
MATCH (p:Patologia)-[:ES_UN*1..3]->(cat:CategoriaSNOMED)
WHERE toLower(p.nombre) CONTAINS toLower($tema)
WITH p.nombre AS patologia, collect(DISTINCT {codigo: cat.codigo, nombre: cat.nombre, nivel: cat.nivel, sistema: cat.sistema}) AS jerarquia
RETURN patologia, jerarquia
ORDER BY patologia
LIMIT 20
""", {"tema": tema})
for r in patologias:
jerarquia_sorted = sorted(r["jerarquia"], key=lambda x: x.get("nivel", 0))
resultado["patologias_snomed"].append({
"nombre": r["patologia"],
"jerarquia": jerarquia_sorted
})
# 3. Anatomía → SNOMED
anatomia = read("""
MATCH (e:EstructuraAnatomica)-[:ES_UN*1..3]->(cat:CategoriaSNOMED)
WHERE toLower(e.nombre) CONTAINS toLower($tema)
WITH e.nombre AS estructura, collect(DISTINCT {codigo: cat.codigo, nombre: cat.nombre, sistema: cat.sistema}) AS jerarquia
RETURN estructura, jerarquia
ORDER BY estructura
LIMIT 20
""", {"tema": tema})
for r in anatomia:
resultado["anatomia_snomed"].append({
"nombre": r["estructura"],
"jerarquia": r["jerarquia"]
})
# 4. Procedimientos → SNOMED
procedimientos = read("""
MATCH (p:Procedimiento)-[:ES_UN*1..3]->(cat:CategoriaSNOMED)
WHERE toLower(p.nombre) CONTAINS toLower($tema)
WITH p.nombre AS procedimiento, collect(DISTINCT {codigo: cat.codigo, nombre: cat.nombre, sistema: cat.sistema}) AS jerarquia
RETURN procedimiento, jerarquia
ORDER BY procedimiento
LIMIT 20
""", {"tema": tema})
for r in procedimientos:
resultado["procedimientos_snomed"].append({
"nombre": r["procedimiento"],
"jerarquia": r["jerarquia"]
})
# 5. Query cruzada: fármacos que tratan patologías del mismo sistema
if resultado["patologias_snomed"]:
sistemas = set()
for p in resultado["patologias_snomed"]:
for j in p["jerarquia"]:
if j.get("sistema"):
sistemas.add(j["sistema"])
for sistema in list(sistemas)[:3]:
cruzada = read("""
MATCH (p:Patologia)-[:ES_UN*1..3]->(sno:CategoriaSNOMED {sistema: $sistema})
WHERE toLower(p.nombre) CONTAINS toLower($tema)
MATCH (p)-[:SE_TRATA_CON]->(f:Farmaco)
OPTIONAL MATCH (f)-[:ES_UN*1..5]->(atc:CategoriaATC)
RETURN DISTINCT f.nombre AS farmaco, p.nombre AS patologia,
collect(DISTINCT atc.nombre)[0] AS clase_atc,
$sistema AS sistema
LIMIT 20
""", {"tema": tema, "sistema": sistema})
for r in cruzada:
resultado["query_cruzada"].append({
"farmaco": r["farmaco"],
"patologia": r["patologia"],
"clase_atc": r.get("clase_atc"),
"sistema_snomed": r["sistema"]
})
resultado["tiene_atc"] = len(resultado["farmacos_atc"]) > 0
resultado["tiene_snomed"] = len(resultado["patologias_snomed"]) > 0 or len(resultado["anatomia_snomed"]) > 0
resultado["tiene_cruzada"] = len(resultado["query_cruzada"]) > 0
return resultado

22
api/routers/pathology.py Normal file
View File

@ -0,0 +1,22 @@
"""Endpoints de patologías."""
from fastapi import APIRouter
from services import graph
router = APIRouter(prefix="/pathology", tags=["pathology"])
@router.get("/{nombre}")
async def get_pathology(nombre: str):
"""Todo sobre una patología: definición, dx, tto, signos, fuentes."""
results = graph.get_pathology(nombre)
if not results:
return {"error": f"Patología '{nombre}' no encontrada"}
return {"results": results, "count": len(results)}
@router.get("/{nombre}/differential")
async def get_differential(nombre: str):
"""Diagnóstico diferencial basado en signos compartidos."""
results = graph.get_differential(nombre)
return {"patologia": nombre, "diferenciales": results, "count": len(results)}

15
api/routers/procedure.py Normal file
View File

@ -0,0 +1,15 @@
"""Endpoints de procedimientos."""
from fastapi import APIRouter
from services import graph
router = APIRouter(prefix="/procedure", tags=["procedure"])
@router.get("/{nombre}")
async def get_procedure(nombre: str):
"""Detalle de un procedimiento: pasos, insumos, parámetros, hallazgos."""
results = graph.get_procedure(nombre)
if not results:
return {"error": f"Procedimiento '{nombre}' no encontrado"}
return {"results": results, "count": len(results)}

42
api/routers/search.py Normal file
View File

@ -0,0 +1,42 @@
"""Endpoints de búsqueda: full-text, semántica e híbrida con RRF."""
from fastapi import APIRouter
from pydantic import BaseModel
from services import vector
router = APIRouter(prefix="/search", tags=["search"])
class SearchRequest(BaseModel):
query: str
top_k: int = 20
libro_id: str | None = None
@router.post("/semantic")
async def semantic_search(req: SearchRequest):
"""Búsqueda por significado usando embeddings (vector KNN)."""
results = vector.search_semantic(req.query, req.top_k, req.libro_id)
return {"query": req.query, "results": results, "count": len(results)}
@router.post("/keyword")
async def keyword_search(req: SearchRequest):
"""Búsqueda full-text con scoring BM25 (Lucene via Neo4j)."""
results = vector.search_keyword(req.query, req.top_k, req.libro_id)
return {"query": req.query, "results": results, "count": len(results)}
@router.post("/hybrid")
async def hybrid_search(req: SearchRequest):
"""Búsqueda híbrida: full-text + semántica fusionadas con Reciprocal Rank Fusion. Usar esta por defecto."""
result = vector.search_hybrid(req.query, req.top_k, req.libro_id)
return {
"query": req.query,
"keyword_count": result["keyword_count"],
"semantic_count": result["semantic_count"],
"intencion": result.get("intencion", "general"),
"query_expandida": result.get("query_expandida", req.query),
"results": result["results"],
"count": len(result["results"])
}

29
api/routers/topics.py Normal file
View File

@ -0,0 +1,29 @@
"""Endpoints de temas y contenido."""
from fastapi import APIRouter
from services import graph
router = APIRouter(prefix="/topics", tags=["topics"])
@router.get("/{up_id}")
async def get_topics(up_id: str):
"""Obtener temas de una UP con fuentes bibliográficas."""
topics = graph.get_topics_by_up(up_id)
return {"up_id": up_id, "topics": topics, "count": len(topics)}
@router.get("/{up_id}/related")
async def get_related(up_id: str):
"""Temas relacionados con otras UPs/materias."""
related = graph.get_related_topics(up_id)
return {"up_id": up_id, "related": related, "count": len(related)}
@router.get("/{tema}/detail")
async def get_detail(tema: str):
"""Detalle completo de un tema: patologías, dx, tto, fuentes."""
detail = graph.get_topic_detail(tema)
if not detail:
return {"error": f"Tema '{tema}' no encontrado"}
return detail

117
api/routers/unified.py Normal file
View File

@ -0,0 +1,117 @@
"""Router inteligente unificado — POST /query.
Un solo endpoint que analiza la pregunta, activa las capas necesarias,
y devuelve un paquete completo de conocimiento.
"""
import asyncio
from fastapi import APIRouter
from pydantic import BaseModel
from services.analyzer import analyze_query
from services.layers import (
execute_ontology,
execute_graph,
execute_bibliography,
execute_activities,
execute_dags,
)
router = APIRouter(tags=["unified"])
LAYER_MAP = {
"ONTOLOGY": ("ontologia", execute_ontology),
"GRAPH": ("grafo", execute_graph),
"BIBLIOGRAPHY": ("bibliografia", None), # special handling for top_k
"ACTIVITIES": ("actividades", execute_activities),
"DAGS": ("dags", execute_dags),
}
class QueryRequest(BaseModel):
pregunta: str
top_k: int = 8
@router.post("/query")
async def unified_query(req: QueryRequest):
"""Consulta inteligente unificada. Analiza la pregunta, detecta entidades,
activa las capas necesarias (ontología, grafo, bibliografía, actividades, DAGs),
y devuelve un paquete completo."""
# 1. Analyzer (Gemini orquestador)
analysis = await asyncio.to_thread(analyze_query, req.pregunta)
# 2. Build task list based on activated layers
tasks = {}
task_keys = []
for layer_code in analysis.get("capas", ["BIBLIOGRAPHY"]):
if layer_code in LAYER_MAP:
key, fn = LAYER_MAP[layer_code]
if layer_code == "BIBLIOGRAPHY":
tasks[key] = asyncio.to_thread(execute_bibliography, analysis, req.top_k)
elif fn:
tasks[key] = asyncio.to_thread(fn, analysis)
task_keys.append(key)
# 3. Execute all layers in parallel
if tasks:
results = await asyncio.gather(*tasks.values(), return_exceptions=True)
else:
results = []
# 4. Compose response
response = {
"pregunta": req.pregunta,
"analisis": {
"intencion": analysis.get("intencion", "general"),
"entidades_detectadas": analysis.get("entidades_detectadas", []),
"capas_activadas": analysis.get("capas", []),
"sub_queries": analysis.get("sub_queries", []),
},
}
for key, result in zip(tasks.keys(), results):
if isinstance(result, Exception):
response[key] = {"error": str(result)}
else:
response[key] = result
# 5. Collect bibliographic sources
fuentes = set()
bib = response.get("bibliografia", {})
if isinstance(bib, dict):
for r in bib.get("results", []):
if isinstance(r, dict) and r.get("libro"):
fuentes.add(r["libro"])
response["fuentes"] = sorted(fuentes)
# 6. Flags
response["tiene_ontologia"] = bool(
response.get("ontologia", {}).get("farmacos_atc")
or response.get("ontologia", {}).get("categorias_encontradas")
or response.get("ontologia", {}).get("entidades_snomed")
)
response["tiene_grafo"] = bool(
response.get("grafo", {}).get("patologias")
or response.get("grafo", {}).get("procedimientos")
or response.get("grafo", {}).get("relaciones")
)
response["tiene_bibliografia"] = bool(
bib.get("results") if isinstance(bib, dict) else False
)
response["tiene_actividades"] = bool(
response.get("actividades", {}).get("actividades")
)
response["tiene_dags"] = bool(
response.get("dags", {}).get("pathways")
or response.get("dags", {}).get("clinical")
)
# 7. Clarificación si Gemini detectó ambigüedad
if analysis.get("ambigua"):
response["necesita_clarificacion"] = True
response["clarificacion"] = analysis.get("clarificacion")
else:
response["necesita_clarificacion"] = False
return response

0
api/services/__init__.py Normal file
View File

344
api/services/analyzer.py Normal file
View File

@ -0,0 +1,344 @@
"""Analyzer inteligente — Gemini 3.1 Flash Lite como orquestador de queries.
Clasifica la pregunta, detecta entidades, y decide qué capas activar.
No busca en Neo4j — solo analiza la pregunta y devuelve un plan de ejecución.
"""
import json
import re
import os
from google import genai
# Reuse existing preprocessing
try:
from services.query import preprocess
except ImportError:
preprocess = None
_client = None
def _get_client():
global _client
if _client is None:
_client = genai.Client(api_key=os.getenv("GCP_API_KEY", ""))
return _client
ANALYZER_PROMPT = """Sos un analizador de queries medicas. Dada una pregunta, devolvé SOLO un JSON válido (sin markdown, sin texto extra).
Formato EXACTO (respetá los valores de tipo y buscar_en tal cual):
{{
"entidades": [
{{"texto": "...", "tipo": "VALOR_TIPO", "buscar_en": "VALOR_LABEL"}}
],
"intencion": "VALOR_INTENCION",
"capas": ["CAPAS"],
"sub_queries": ["reformulacion 1", "reformulacion 2", "reformulacion 3"],
"ambigua": false,
"clarificacion": null
}}
Si la pregunta es AMBIGUA o DEMASIADO AMPLIA (ej: "resumen de todo el curso", "todo sobre farmacología", "preparame para el examen"), setear:
"ambigua": true,
"clarificacion": {{"pregunta": "pregunta para el usuario", "opciones": ["opcion 1", "opcion 2", "opcion 3"]}}
Igualmente, SIEMPRE generar entidades, sub_queries y capas con la mejor interpretación posible. La clarificación es un COMPLEMENTO, no un reemplazo de la búsqueda.
Valores EXACTOS permitidos para tipo y buscar_en:
- tipo=patologia, buscar_en=Patologia (enfermedades: otitis, neumonía, diabetes, HTA, psoriasis)
- tipo=farmaco, buscar_en=Farmaco (fármacos específicos: furosemida, amoxicilina, ibuprofeno, metformina)
- tipo=clase_farmacologica, buscar_en=CategoriaATC (clases de fármacos: diuréticos, betalactámicos, AINEs, corticoides, opioides)
- tipo=sistema_corporal, buscar_en=CategoriaSNOMED (sistemas: cardiovascular, respiratorio, nervioso, digestivo)
- tipo=anatomia, buscar_en=EstructuraAnatomica (estructuras: oído medio, retina, riñón, hígado, membrana timpánica)
- tipo=procedimiento, buscar_en=Procedimiento (técnicas: otoscopía, ECG, Rx tórax, toma de TA)
- tipo=signo, buscar_en=Signo (signos clínicos: fiebre, edema, soplo, cianosis)
- tipo=sintoma, buscar_en=Sintoma (síntomas: otalgia, cefalea, disnea, dolor torácico)
- tipo=metodo_dx, buscar_en=MetodoDx (métodos diagnósticos: hemograma, ecografía, audiometría)
- tipo=actividad_academica, buscar_en=Actividad (TPs, seminarios, talleres, guías, contenidos UP)
Valores para intencion: tratamiento|diagnostico|anatomia|fisiologia|clinica|etiologia|clasificacion|procedimiento|actividad|bibliografia|general
Valores para capas: ONTOLOGY, GRAPH, BIBLIOGRAPHY, ACTIVITIES, DAGS
Reglas para decidir capas:
- BIBLIOGRAPHY y ACTIVITIES siempre incluir
- Si hay patologia, farmaco, clase_farmacologica o sistema_corporal → agregar ONTOLOGY y GRAPH
- Si hay signo, sintoma, procedimiento, metodo_dx o anatomia → agregar ONTOLOGY y GRAPH (procedimientos, signos y anatomía también están clasificados en SNOMED)
- Si pregunta "qué hago ante", "cómo manejar", "paciente con" → agregar DAGS
- Si pregunta "fisiopatología", "mecanismo", "por qué se produce" → agregar DAGS
- Si pregunta "clasificación", "qué tipo", "a qué clase pertenece" → agregar ONTOLOGY
sub_queries: genera 3 reformulaciones especializadas para búsqueda en textos médicos, cubriendo diferentes aspectos del tema (definición, clínica, tratamiento, etc.)
EJEMPLOS:
Q: "qué diuréticos se usan en insuficiencia cardíaca"
{{"entidades":[{{"texto":"diuréticos","tipo":"clase_farmacologica","buscar_en":"CategoriaATC"}},{{"texto":"insuficiencia cardíaca","tipo":"patologia","buscar_en":"Patologia"}}],"intencion":"tratamiento","capas":["ONTOLOGY","GRAPH","BIBLIOGRAPHY","ACTIVITIES"],"sub_queries":["diuréticos tratamiento insuficiencia cardíaca","furosemida hidroclorotiazida espironolactona IC dosis","insuficiencia cardíaca manejo farmacológico guías"]}}
Q: "preparame para el TP de otoscopía"
{{"entidades":[{{"texto":"TP otoscopía","tipo":"actividad_academica","buscar_en":"Actividad"}},{{"texto":"otoscopía","tipo":"procedimiento","buscar_en":"Procedimiento"}}],"intencion":"procedimiento","capas":["ACTIVITIES","GRAPH","ONTOLOGY","BIBLIOGRAPHY"],"sub_queries":["otoscopía técnica pasos procedimiento","guía TP otoscopía lista cotejo","membrana timpánica hallazgos normales patológicos otoscopía"]}}
Q: "paciente con otalgia y fiebre qué hago"
{{"entidades":[{{"texto":"otalgia","tipo":"sintoma","buscar_en":"Sintoma"}},{{"texto":"fiebre","tipo":"signo","buscar_en":"Signo"}}],"intencion":"clinica","capas":["DAGS","GRAPH","ONTOLOGY","BIBLIOGRAPHY","ACTIVITIES"],"sub_queries":["otalgia fiebre diagnóstico diferencial","otitis media aguda diagnóstico tratamiento","manejo clínico otalgia aguda evaluación otoscópica"]}}
Q: "contenidos unidad 1"
{{"entidades":[{{"texto":"unidad 1","tipo":"actividad_academica","buscar_en":"Actividad"}},{{"texto":"contenidos","tipo":"actividad_academica","buscar_en":"Actividad"}}],"intencion":"actividad","capas":["ACTIVITIES","BIBLIOGRAPHY"],"sub_queries":["contenidos unidad 1 temas unidad","guía contenidos first unit","unit 1 core topics fundamentals introduction"],"ambigua":false,"clarificacion":null}}
Q: "resumen completo del curso"
{{"entidades":[{{"texto":"course","tipo":"actividad_academica","buscar_en":"Actividad"}}],"intencion":"actividad","capas":["ACTIVITIES","BIBLIOGRAPHY"],"sub_queries":["contenidos curso temas unidades","guía contenidos del curso"],"ambigua":true,"clarificacion":{{"pregunta":"The course has multiple units. ¿Sobre cuál querés el resumen?","opciones":["Unit 1: Introduction and fundamentals","Unit 2: Core concepts","Unit 3: Advanced topics","Unit 4: Applied knowledge","All units"]}}}}
Pregunta: {pregunta}"""
# Normalización post-Gemini: mapea tipos genéricos a los del sistema
_TYPE_NORMALIZE = {
# Fármacos
"medicamento": ("farmaco", "Farmaco"),
"medicina": ("farmaco", "Farmaco"),
"droga": ("farmaco", "Farmaco"),
"drug": ("farmaco", "Farmaco"),
"farmaco": ("farmaco", "Farmaco"),
"fármaco": ("farmaco", "Farmaco"),
# Clases farmacológicas
"clase de farmaco": ("clase_farmacologica", "CategoriaATC"),
"clase farmacologica": ("clase_farmacologica", "CategoriaATC"),
"grupo farmacologico": ("clase_farmacologica", "CategoriaATC"),
"clase_farmacologica": ("clase_farmacologica", "CategoriaATC"),
# Patologías
"enfermedad": ("patologia", "Patologia"),
"patologia": ("patologia", "Patologia"),
"patología": ("patologia", "Patologia"),
"condicion": ("patologia", "Patologia"),
"trastorno": ("patologia", "Patologia"),
"sindrome": ("patologia", "Patologia"),
"disease": ("patologia", "Patologia"),
# Anatomía
"organo": ("anatomia", "EstructuraAnatomica"),
"órgano": ("anatomia", "EstructuraAnatomica"),
"estructura": ("anatomia", "EstructuraAnatomica"),
"anatomia": ("anatomia", "EstructuraAnatomica"),
"anatomía": ("anatomia", "EstructuraAnatomica"),
"estructura_anatomica": ("anatomia", "EstructuraAnatomica"),
# Sistemas
"sistema": ("sistema_corporal", "CategoriaSNOMED"),
"sistema_corporal": ("sistema_corporal", "CategoriaSNOMED"),
"aparato": ("sistema_corporal", "CategoriaSNOMED"),
# Procedimientos
"procedimiento": ("procedimiento", "Procedimiento"),
"tecnica": ("procedimiento", "Procedimiento"),
"estudio": ("procedimiento", "Procedimiento"),
# Signos y síntomas
"signo": ("signo", "Signo"),
"sintoma": ("sintoma", "Sintoma"),
"síntoma": ("sintoma", "Sintoma"),
"hallazgo": ("signo", "Signo"),
# Diagnóstico
"metodo_dx": ("metodo_dx", "MetodoDx"),
"metodo diagnostico": ("metodo_dx", "MetodoDx"),
"prueba": ("metodo_dx", "MetodoDx"),
"test": ("metodo_dx", "MetodoDx"),
# Actividades
"actividad_academica": ("actividad_academica", "Actividad"),
"actividad": ("actividad_academica", "Actividad"),
"tp": ("actividad_academica", "Actividad"),
"documento": ("actividad_academica", "Actividad"),
# Tipos genéricos que Gemini inventa
"agente": ("patologia", "Agente"),
"agente_infeccioso": ("patologia", "Agente"),
"microorganismo": ("patologia", "Agente"),
"virus": ("patologia", "Agente"),
"bacteria": ("patologia", "Agente"),
"parasito": ("patologia", "Agente"),
"parametro": ("metodo_dx", "Parametro"),
"valor": ("metodo_dx", "Parametro"),
"laboratorio": ("metodo_dx", "MetodoDx"),
"hallazgo": ("signo", "Hallazgo"),
"grupo_farmacologico": ("clase_farmacologica", "GrupoFarmacologico"),
"clase": ("clase_farmacologica", "CategoriaATC"),
"categoria": ("clase_farmacologica", "CategoriaATC"),
"tratamiento": ("farmaco", "Farmaco"),
"cirugia": ("procedimiento", "Procedimiento"),
"cirugía": ("procedimiento", "Procedimiento"),
"imagen": ("metodo_dx", "MetodoDx"),
"radiografia": ("metodo_dx", "MetodoDx"),
"ecografia": ("metodo_dx", "MetodoDx"),
"analisis": ("metodo_dx", "MetodoDx"),
}
_BUSCAR_EN_NORMALIZE = {
"medicamentos": "Farmaco",
"enfermedades": "Patologia",
"farmacos": "Farmaco",
"organos": "EstructuraAnatomica",
"estructuras": "EstructuraAnatomica",
"procedimientos": "Procedimiento",
"signos": "Signo",
"sintomas": "Sintoma",
"sistemas": "CategoriaSNOMED",
"actividades": "Actividad",
"patologias": "Patologia",
}
def _normalize_entities(entities: list) -> list:
"""Normaliza tipos genéricos de Gemini a los labels exactos del sistema."""
for ent in entities:
tipo = ent.get("tipo", "").lower().strip()
buscar = ent.get("buscar_en", "").strip()
# Normalizar tipo
if tipo in _TYPE_NORMALIZE:
ent["tipo"], ent["buscar_en"] = _TYPE_NORMALIZE[tipo]
# Normalizar buscar_en si Gemini devolvió algo genérico
buscar_lower = buscar.lower()
if buscar_lower in _BUSCAR_EN_NORMALIZE:
ent["buscar_en"] = _BUSCAR_EN_NORMALIZE[buscar_lower]
return entities
def analyze_query(pregunta: str) -> dict:
"""Analiza una pregunta médica y devuelve plan de ejecución."""
# 1. Preprocessing existente (sinónimos, intención básica)
pre = None
if preprocess:
try:
pre = preprocess(pregunta)
except Exception:
pass
expandida = pre.get("expandida", pregunta) if pre else pregunta
# 2. Gemini clasifica via Google GenAI
client = _get_client()
try:
resp = client.models.generate_content(
model="gemini-2.5-flash",
contents=ANALYZER_PROMPT.format(pregunta=pregunta),
config={
"temperature": 0.1,
"max_output_tokens": 4096,
"response_mime_type": "application/json",
},
)
raw_text = resp.text
# Log raw for debugging
import logging
logging.info(f"Gemini raw ({len(raw_text)} chars): {raw_text[:200]}")
# Limpiar markdown
text = raw_text.strip()
if "```" in text:
text = re.sub(r'```\w*\n?', '', text)
text = text.strip()
# Extraer JSON: buscar primer { y último }
first_brace = text.find('{')
last_brace = text.rfind('}')
if first_brace >= 0 and last_brace > first_brace:
text = text[first_brace:last_brace+1]
result = json.loads(text)
# Normalizar keys (Gemini puede usar nombres diferentes)
if "entidades" not in result:
# Buscar keys alternativas
for alt_key in ["entities", "entidad", "detected_entities"]:
if alt_key in result:
result["entidades"] = result.pop(alt_key)
break
else:
result["entidades"] = []
if "capas" not in result:
for alt_key in ["layers", "capa", "activated_layers"]:
if alt_key in result:
result["capas"] = result.pop(alt_key)
break
else:
result["capas"] = ["BIBLIOGRAPHY"]
if "intencion" not in result:
for alt_key in ["intent", "intention"]:
if alt_key in result:
result["intencion"] = result.pop(alt_key)
break
else:
result["intencion"] = "general"
if "sub_queries" not in result:
result["sub_queries"] = [pregunta]
# Normalizar tipos genéricos de Gemini a labels del sistema
if result.get("entidades"):
result["entidades"] = _normalize_entities(result["entidades"])
except Exception as e:
# Log the error AND the raw response for debugging
import logging
try:
raw = response.text[:300] if 'response' in dir() and hasattr(response, 'text') else 'no response'
except:
raw = 'no response'
logging.error(f"Gemini analyzer failed: {type(e).__name__}: {str(e)[:200]} | Raw: {raw}")
# Fallback determinista
result = _fallback_analysis(pregunta, pre)
# 3. Combinar con preprocessing existente
analysis = {
"original": pregunta,
"intencion": result.get("intencion", "general"),
"expandida": expandida,
"sub_queries": result.get("sub_queries", [pregunta]),
"entidades_detectadas": result.get("entidades", []),
"capas": result.get("capas", ["BIBLIOGRAPHY"]),
}
# Propagar clarificación si Gemini detectó ambigüedad
if result.get("ambigua"):
analysis["ambigua"] = True
analysis["clarificacion"] = result.get("clarificacion")
# BIBLIOGRAPHY y ACTIVITIES siempre activas (somos un sistema para estudiantes)
for capa_base in ["BIBLIOGRAPHY", "ACTIVITIES"]:
if capa_base not in analysis["capas"]:
analysis["capas"].append(capa_base)
return analysis
def _fallback_analysis(pregunta: str, pre: dict = None) -> dict:
"""Análisis determinista como fallback si Gemini falla."""
pregunta_lower = pregunta.lower()
capas = ["BIBLIOGRAPHY", "ACTIVITIES"]
entidades = []
intencion = pre.get("intencion", "general") if pre else "general"
# Detectar actividades
activity_keywords = ["tp", "seminario", "taller", "acreditación", "acreditacion", "guía", "guia", "lista de cotejo"]
if any(kw in pregunta_lower for kw in activity_keywords):
capas.append("ACTIVITIES")
entidades.append({"texto": pregunta, "tipo": "actividad_academica", "buscar_en": "Actividad"})
# Detectar intención clínica
clinical_keywords = ["qué hago", "que hago", "cómo manejar", "como manejar", "ante un paciente", "manejo de"]
if any(kw in pregunta_lower for kw in clinical_keywords):
capas.append("DAGS")
# Detectar intención ontológica
onto_keywords = ["clasificación", "clasificacion", "qué tipo", "que tipo", "a qué clase", "pertenece"]
if any(kw in pregunta_lower for kw in onto_keywords):
capas.append("ONTOLOGY")
# Default: siempre grafo
if "GRAPH" not in capas:
capas.append("GRAPH")
return {
"entidades": entidades,
"intencion": intencion,
"capas": list(set(capas)),
"sub_queries": [pregunta],
}

307
api/services/graph.py Normal file
View File

@ -0,0 +1,307 @@
"""Servicio de consultas al grafo Neo4j. Operaciones predefinidas, no queries abiertas."""
import os
import logging
import threading
import time
from neo4j import GraphDatabase
from neo4j.exceptions import ServiceUnavailable, SessionExpired
from dotenv import load_dotenv
load_dotenv()
URI = os.getenv("NEO4J_URI")
USER = os.getenv("NEO4J_USERNAME")
PASSWORD = os.getenv("NEO4J_PASSWORD")
DATABASE = os.getenv("NEO4J_DATABASE")
_keepalive_started = False
_driver = None
def _keepalive_loop():
"""Thread que mantiene la conexión a Neo4j viva con un ping cada 45 segundos."""
while True:
time.sleep(45)
try:
driver = get_driver()
with driver.session(database=DATABASE) as session:
session.run("RETURN 1")
except Exception as e:
logging.warning(f"Keep-alive ping failed: {e}")
_reset_driver()
def get_driver():
global _driver, _keepalive_started
if _driver is None:
_driver = GraphDatabase.driver(
URI, auth=(USER, PASSWORD),
max_connection_lifetime=300,
max_connection_pool_size=10,
connection_acquisition_timeout=30,
connection_timeout=15,
)
if not _keepalive_started:
t = threading.Thread(target=_keepalive_loop, daemon=True)
t.start()
_keepalive_started = True
logging.info("Neo4j keep-alive thread started (45s interval)")
return _driver
def _reset_driver():
"""Fuerza recrear el driver si la conexión se perdió."""
global _driver
if _driver:
try:
_driver.close()
except Exception:
pass
_driver = None
def query(cypher: str, params: dict = None, retries: int = 2) -> list:
for attempt in range(retries + 1):
try:
with get_driver().session(database=DATABASE) as session:
result = session.run(cypher, params or {})
return [record.data() for record in result]
except (ServiceUnavailable, SessionExpired, OSError) as e:
logging.warning(f"Neo4j query retry {attempt+1}/{retries+1}: {e}")
_reset_driver()
if attempt == retries:
raise
def write(cypher: str, params: dict = None, retries: int = 2):
"""Ejecuta una query de escritura con retry."""
for attempt in range(retries + 1):
try:
with get_driver().session(database=DATABASE) as session:
session.execute_write(lambda tx: tx.run(cypher, params or {}))
return
except (ServiceUnavailable, SessionExpired, OSError) as e:
logging.warning(f"Neo4j write retry {attempt+1}/{retries+1}: {e}")
_reset_driver()
if attempt == retries:
raise
# === TOPICS ===
def get_topics_by_up(up_id: str) -> list:
return query("""
MATCH (t:Tema)-[:PERTENECE_A]->(up:UP {id: $up_id})
OPTIONAL MATCH (t)-[:CONTENIDO_EN]->(f:Fuente)
OPTIONAL MATCH (t)-[:AREA_DE]->(e:Especialidad)
RETURN t.nombre AS tema, t.descripcion AS descripcion,
e.nombre AS especialidad,
collect(DISTINCT {titulo: f.titulo, id: f.id}) AS fuentes
ORDER BY e.nombre, t.nombre
""", {"up_id": up_id})
def get_related_topics(up_id: str) -> list:
return query("""
MATCH (up1:UP {id: $up_id})<-[:PERTENECE_A]-(t1:Tema)-[:TRATA]->(concepto)
MATCH (concepto)<-[:TRATA]-(t2:Tema)-[:PERTENECE_A]->(up2:UP)
WHERE up1 <> up2
RETURN t1.nombre AS tema_origen, concepto.nombre AS concepto_compartido,
t2.nombre AS tema_relacionado, up2.id AS up_relacionada, up2.nombre AS up_nombre
""", {"up_id": up_id})
def get_topic_detail(tema_nombre: str) -> dict:
results = query("""
MATCH (t:Tema {nombre: $nombre})
OPTIONAL MATCH (t)-[:TRATA]->(p:Patologia)
OPTIONAL MATCH (p)-[:SE_DIAGNOSTICA_CON]->(dx:MetodoDx)
OPTIONAL MATCH (p)-[:SE_TRATA_CON]->(tx)
OPTIONAL MATCH (p)-[:PRESENTA]->(s:Signo)
OPTIONAL MATCH (t)-[:CONTENIDO_EN]->(f:Fuente)
OPTIONAL MATCH (t)-[:INCLUYE]->(proc:Procedimiento)
RETURN t.nombre AS tema, t.descripcion AS descripcion,
collect(DISTINCT {nombre: p.nombre, definicion: p.definicion, cie10: p.cie10}) AS patologias,
collect(DISTINCT dx.nombre) AS metodos_dx,
collect(DISTINCT s.nombre) AS signos,
collect(DISTINCT {titulo: f.titulo, id: f.id}) AS fuentes,
collect(DISTINCT proc.nombre) AS procedimientos
""", {"nombre": tema_nombre})
return results[0] if results else None
# === PATHOLOGY ===
def get_pathology(nombre: str) -> list:
return query("""
MATCH (p:Patologia)
WHERE toLower(p.nombre) CONTAINS toLower($nombre)
OPTIONAL MATCH (p)-[:SE_DIAGNOSTICA_CON]->(dx:MetodoDx)
OPTIONAL MATCH (p)-[:SE_TRATA_CON]->(tx)
OPTIONAL MATCH (p)-[:PRESENTA]->(s:Signo)
OPTIONAL MATCH (p)-[:PREVALENTE_EN]->(ge:GrupoEtario)
OPTIONAL MATCH (t:Tema)-[:TRATA]->(p)
OPTIONAL MATCH (t)-[:CONTENIDO_EN]->(f:Fuente)
RETURN p.nombre AS nombre, p.definicion AS definicion,
p.cie10 AS cie10, p.via_transmision AS via_transmision,
collect(DISTINCT dx.nombre) AS metodos_dx,
collect(DISTINCT s.nombre) AS signos,
collect(DISTINCT {titulo: f.titulo, id: f.id}) AS fuentes,
collect(DISTINCT ge.nombre) AS grupos_etarios,
collect(DISTINCT t.nombre) AS temas
""", {"nombre": nombre})
def get_differential(nombre: str) -> list:
return query("""
MATCH (p:Patologia)
WHERE toLower(p.nombre) CONTAINS toLower($nombre)
MATCH (p)-[:PRESENTA]->(s:Signo)<-[:PRESENTA]-(other:Patologia)
WHERE p <> other
RETURN other.nombre AS patologia, other.definicion AS definicion,
collect(DISTINCT s.nombre) AS signos_compartidos
ORDER BY size(collect(DISTINCT s.nombre)) DESC
""", {"nombre": nombre})
# === PROCEDURES ===
def get_procedure(nombre: str) -> list:
return query("""
MATCH (proc:Procedimiento)
WHERE toLower(proc.nombre) CONTAINS toLower($nombre)
OPTIONAL MATCH (proc)-[:EVALUA]->(param:Parametro)
OPTIONAL MATCH (proc)-[:PUEDE_DETECTAR]->(hall:Signo)
OPTIONAL MATCH (proc)-[:EVALUA_PUNTO]->(punto:Signo)
OPTIONAL MATCH (proc)-[:EVALUA_GANGLIO]->(gang:Signo)
RETURN proc.nombre AS nombre, proc.definicion AS definicion,
proc.indicaciones AS indicaciones, proc.insumos AS insumos,
proc.pasos_totales AS pasos_totales, proc.fuente_gus AS fuente_gus,
collect(DISTINCT {nombre: param.nombre, normal: param.valores_normales}) AS parametros,
collect(DISTINCT hall.nombre) AS hallazgos_detectables,
collect(DISTINCT punto.nombre) AS puntos_dolorosos,
collect(DISTINCT gang.nombre) AS ganglios
""", {"nombre": nombre})
# === STATS ===
# === ACTIVITIES ===
def get_activity(nombre: str) -> list:
"""Busca actividades (TP, Seminario, Taller, Acreditacion) por nombre, titulo o temas.
Busca por frase completa y por palabras individuales (>= 4 chars) para
maximizar matches. Ej: "rinoscopia anterior" matchea titulo "Rinoscopia".
"""
# Primero intento con la frase completa
results = query("""
MATCH (a:Actividad)
WHERE toLower(a.nombre) CONTAINS toLower($nombre)
OR toLower(a.titulo) CONTAINS toLower($nombre)
OR toLower(a.id) CONTAINS toLower($nombre)
OPTIONAL MATCH (a)-[:ABORDA]->(t:Tema)
OPTIONAL MATCH (a)-[:PRACTICA]->(proc:Procedimiento)
OPTIONAL MATCH (a)-[:TIENE_DOCUMENTO]->(d:Documento)
OPTIONAL MATCH (a)-[:PERTENECE_A]->(up:UP)
RETURN a.id AS id, a.nombre AS nombre, a.titulo AS titulo, a.tipo AS tipo,
up.nombre AS unidad_problematica, up.id AS up_id,
collect(DISTINCT t.nombre) AS temas,
collect(DISTINCT proc.nombre) AS procedimientos,
collect(DISTINCT {id: d.id, nombre: d.nombre, tipo: d.tipo, archivo: d.archivo}) AS documentos
""", {"nombre": nombre})
if results:
return results
# Fallback: buscar por cada palabra individual (>= 4 chars)
words = [w for w in nombre.lower().split() if len(w) >= 4]
if not words:
return []
# Buscar actividades cuyo titulo contenga alguna de las palabras
for word in words:
results = query("""
MATCH (a:Actividad)
WHERE toLower(a.titulo) CONTAINS toLower($word)
OR toLower(a.nombre) CONTAINS toLower($word)
OPTIONAL MATCH (a)-[:ABORDA]->(t:Tema)
OPTIONAL MATCH (a)-[:PRACTICA]->(proc:Procedimiento)
OPTIONAL MATCH (a)-[:TIENE_DOCUMENTO]->(d:Documento)
OPTIONAL MATCH (a)-[:PERTENECE_A]->(up:UP)
RETURN a.id AS id, a.nombre AS nombre, a.titulo AS titulo, a.tipo AS tipo,
up.nombre AS unidad_problematica, up.id AS up_id,
collect(DISTINCT t.nombre) AS temas,
collect(DISTINCT proc.nombre) AS procedimientos,
collect(DISTINCT {id: d.id, nombre: d.nombre, tipo: d.tipo, archivo: d.archivo}) AS documentos
""", {"word": word})
if results:
return results
# Último fallback: buscar en temas que aborda la actividad
for word in words:
results = query("""
MATCH (a:Actividad)-[:ABORDA]->(t:Tema)
WHERE toLower(t.nombre) CONTAINS toLower($word)
OPTIONAL MATCH (a)-[:PRACTICA]->(proc:Procedimiento)
OPTIONAL MATCH (a)-[:TIENE_DOCUMENTO]->(d:Documento)
OPTIONAL MATCH (a)-[:PERTENECE_A]->(up:UP)
RETURN a.id AS id, a.nombre AS nombre, a.titulo AS titulo, a.tipo AS tipo,
up.nombre AS unidad_problematica, up.id AS up_id,
collect(DISTINCT t.nombre) AS temas,
collect(DISTINCT proc.nombre) AS procedimientos,
collect(DISTINCT {id: d.id, nombre: d.nombre, tipo: d.tipo, archivo: d.archivo}) AS documentos
""", {"word": word})
if results:
return results
return []
def list_activities(up_id: str = None, tipo: str = None) -> list:
"""Lista actividades filtradas por UP y/o tipo."""
where_clauses = []
params = {}
if up_id:
where_clauses.append("up.id = $up_id")
params["up_id"] = up_id
if tipo:
where_clauses.append("a.tipo = $tipo")
params["tipo"] = tipo
where = "WHERE " + " AND ".join(where_clauses) if where_clauses else ""
return query(f"""
MATCH (a:Actividad)-[:PERTENECE_A]->(up:UP)
{where}
OPTIONAL MATCH (a)-[:ABORDA]->(t:Tema)
RETURN a.id AS id, a.nombre AS nombre, a.titulo AS titulo, a.tipo AS tipo,
up.id AS up_id, collect(DISTINCT t.nombre) AS temas
ORDER BY a.nombre
""", params)
def get_activity_material(activity_id: str) -> list:
"""Devuelve el contenido completo de los documentos vinculados a una actividad."""
return query("""
MATCH (a:Actividad {id: $id})-[:TIENE_DOCUMENTO]->(d:Documento)
RETURN d.id AS doc_id, d.nombre AS nombre, d.tipo AS tipo,
d.archivo AS archivo, d.texto AS texto, d.palabras AS palabras
ORDER BY d.tipo, d.nombre
""", {"id": activity_id})
# === STATS ===
def get_stats() -> dict:
nodos = query("MATCH (n) RETURN labels(n)[0] AS tipo, count(n) AS cantidad ORDER BY cantidad DESC")
rels = query("MATCH ()-[r]->() RETURN type(r) AS tipo, count(r) AS cantidad ORDER BY cantidad DESC")
return {
"total_nodos": sum(n["cantidad"] for n in nodos),
"total_relaciones": sum(r["cantidad"] for r in rels),
"nodos": {n["tipo"]: n["cantidad"] for n in nodos},
"relaciones": {r["tipo"]: r["cantidad"] for r in rels}
}

362
api/services/ingest.py Normal file
View File

@ -0,0 +1,362 @@
"""Servicio de ingesta de libros: PDF -> parse -> upload -> vectorize.
Pipeline simplificado para correr dentro de Cloud Run.
Para libros grandes (>1000 pags), preferir correr desde CLI local.
"""
import os
import re
import json
import time
import unicodedata
import fitz # PyMuPDF
from services.graph import query, write
# Chunking config
TARGET_SIZE = 280
MIN_SIZE = 150
MAX_SIZE = 380
OVERLAP_SIZE = 60
PARENT_WINDOW = 3
MAX_PARENT_WORDS = 1200
BATCH_SIZE = 100
# Jobs in memory (single instance is fine for this use case)
_jobs = {}
def get_job(job_id: str) -> dict:
return _jobs.get(job_id)
def normalize_for_search(text: str) -> str:
if not text:
return ""
nfkd = unicodedata.normalize('NFKD', text)
return ''.join(c for c in nfkd if not unicodedata.combining(c)).lower()
def clean_text(text: str) -> str:
text = re.sub(r'\n{3,}', '\n\n', text)
text = re.sub(r'^\s*\d{1,4}\s*$', '', text, flags=re.MULTILINE)
text = re.sub(r'(?i)^.*booksmedicos\.org.*$', '', text, flags=re.MULTILINE)
text = re.sub(r'[ \t]+', ' ', text)
text = '\n'.join(line.strip() for line in text.split('\n'))
return text.strip()
def classify_content_type(text: str) -> str:
lines = text.strip().split('\n')
if not lines:
return "body"
tab_lines = sum(1 for l in lines if '|' in l or '\t' in l)
if tab_lines > len(lines) * 0.3 and tab_lines >= 3:
return "tabla"
list_lines = sum(1 for l in lines if re.match(r'^\s*[-*]\s', l) or re.match(r'^\s*\d+[\.\)]\s', l))
if list_lines > len(lines) * 0.3 and list_lines >= 3:
return "lista"
first_100 = text[:200].lower()
if any(p in first_100 for p in ['concepto', 'definicion', 'se define como']):
return "definicion"
return "body"
def _find_sentence_boundary(words, target_idx):
search_start = max(0, target_idx - 30)
search_end = min(len(words), target_idx + 30)
best = target_idx
best_dist = 999
for i in range(search_start, search_end):
word = words[i]
if word.endswith(('.', '?', '!', ':')) and not re.match(r'^\d+\.$', word):
dist = abs(i - target_idx)
if dist < best_dist:
best = i
best_dist = dist
return best + 1
def parse_and_chunk(pdf_path: str, libro_id: str) -> tuple:
"""Parse PDF and generate v2 chunks. Returns (children, parents)."""
doc = fitz.open(pdf_path)
pages = []
for i in range(doc.page_count):
text = clean_text(doc[i].get_text())
if len(text.strip()) > 20:
pages.append({"page": i + 1, "text": text})
doc.close()
# Simple structure detection (generic patterns)
cap_pats = [re.compile(p, re.MULTILINE) for p in [
r'^SECCION\s+[IVXLCDM]+', r'^Capitulo\s+\d+', r'^PARTE\s+\d+',
]]
sec_pats = [re.compile(p, re.MULTILINE) for p in [
r'^\d+\.\d+[\s\.]+[A-Z]', r'^[A-Z][A-Z\s]{5,60}$',
]]
current_cap = ""
current_sec = ""
structured = []
for p in pages:
for line in p["text"].split('\n'):
ls = line.strip()
if not ls or len(ls) < 3:
continue
is_cap = any(pat.match(ls) for pat in cap_pats)
if is_cap:
current_cap = ls[:120]
current_sec = ""
elif any(pat.match(ls) for pat in sec_pats) and len(ls) < 80:
current_sec = ls[:100]
structured.append({**p, "titulo_capitulo": current_cap, "titulo_seccion": current_sec})
# Generate chunks
all_words = []
word_meta = []
for sp in structured:
for w in sp["text"].split():
all_words.append(w)
word_meta.append((sp["page"], sp["titulo_capitulo"], sp["titulo_seccion"]))
if not all_words:
return [], []
children = []
pos = 0
chunk_index = 0
while pos < len(all_words):
end_target = pos + TARGET_SIZE
if end_target >= len(all_words):
end = len(all_words)
else:
end = _find_sentence_boundary(all_words, end_target)
if end - pos > MAX_SIZE:
end = pos + MAX_SIZE
if end - pos < MIN_SIZE and children:
prev = children[-1]
prev["text"] = prev["text"] + " " + " ".join(all_words[pos:end])
prev["word_count"] = len(prev["text"].split())
prev["page_end"] = word_meta[end - 1][0]
break
chunk_text = " ".join(all_words[pos:end])
capitulo = word_meta[pos][1]
seccion = word_meta[pos][2]
for i in range(pos, min(end, pos + 50)):
if word_meta[i][1]:
capitulo = word_meta[i][1]
if word_meta[i][2]:
seccion = word_meta[i][2]
text_busqueda = normalize_for_search(chunk_text)
kw_parts = [capitulo, seccion, " ".join(chunk_text.split()[:50])]
keywords = normalize_for_search(" ".join(p for p in kw_parts if p))
children.append({
"id": f"{libro_id}_v2_{chunk_index:05d}",
"libro_id": libro_id,
"page_start": word_meta[pos][0],
"page_end": word_meta[end - 1][0],
"text": chunk_text,
"word_count": end - pos,
"titulo_capitulo": capitulo,
"titulo_seccion": seccion,
"tipo_contenido": classify_content_type(chunk_text),
"parent_id": "",
"chunk_index": chunk_index,
"version": 2,
"text_busqueda": text_busqueda,
"titulo_seccion_busqueda": normalize_for_search(seccion),
"titulo_capitulo_busqueda": normalize_for_search(capitulo),
"keywords": keywords,
})
chunk_index += 1
next_pos = end - OVERLAP_SIZE
if next_pos <= pos:
next_pos = end
pos = next_pos
# Parents
parents = []
for i in range(0, len(children), PARENT_WINDOW):
window = children[i:i + PARENT_WINDOW]
parent_text = " ".join(c["text"] for c in window)
if len(parent_text.split()) > MAX_PARENT_WORDS:
parent_text = " ".join(parent_text.split()[:MAX_PARENT_WORDS])
parent_id = f"{libro_id}_v2_parent_{len(parents):05d}"
parents.append({
"id": parent_id,
"libro_id": libro_id,
"page_start": window[0]["page_start"],
"page_end": window[-1]["page_end"],
"text": parent_text,
"word_count": len(parent_text.split()),
"titulo_capitulo": window[0]["titulo_capitulo"],
"titulo_seccion": window[0]["titulo_seccion"],
})
for c in window:
c["parent_id"] = parent_id
return children, parents
def upload_to_neo4j(libro_id: str, children: list, parents: list):
"""Upload chunks and parents to Neo4j, replacing existing ones for this libro."""
# Delete existing
write("MATCH (c:Chunk {libro_id: $lid}) DETACH DELETE c", {"lid": libro_id})
write("MATCH (p:ParentChunk {libro_id: $lid}) DETACH DELETE p", {"lid": libro_id})
# Upload children in batches
for i in range(0, len(children), BATCH_SIZE):
batch = children[i:i + BATCH_SIZE]
write("""
UNWIND $batch AS c
CREATE (n:Chunk {
id: c.id, libro_id: c.libro_id, text: c.text,
page_start: c.page_start, page_end: c.page_end,
word_count: c.word_count, titulo_capitulo: c.titulo_capitulo,
titulo_seccion: c.titulo_seccion, tipo_contenido: c.tipo_contenido,
parent_id: c.parent_id, chunk_index: c.chunk_index, version: c.version,
text_busqueda: c.text_busqueda, titulo_seccion_busqueda: c.titulo_seccion_busqueda,
titulo_capitulo_busqueda: c.titulo_capitulo_busqueda, keywords: c.keywords
})
""", {"batch": batch})
# Upload parents
for i in range(0, len(parents), BATCH_SIZE):
batch = parents[i:i + BATCH_SIZE]
write("""
UNWIND $batch AS p
CREATE (n:ParentChunk {
id: p.id, libro_id: p.libro_id, text: p.text,
page_start: p.page_start, page_end: p.page_end,
word_count: p.word_count, titulo_capitulo: p.titulo_capitulo,
titulo_seccion: p.titulo_seccion
})
""", {"batch": batch})
# Create CHILD_OF relationships
write("""
MATCH (child:Chunk {libro_id: $lid})
WHERE child.parent_id IS NOT NULL AND child.parent_id <> ''
WITH child
MATCH (parent:ParentChunk {id: child.parent_id})
CREATE (child)-[:CHILD_OF]->(parent)
""", {"lid": libro_id})
# Create SIGUE_A relationships
write("""
MATCH (c1:Chunk {libro_id: $lid})
WITH c1 ORDER BY c1.chunk_index
WITH collect(c1) AS chunks
UNWIND range(0, size(chunks)-2) AS i
WITH chunks[i] AS c1, chunks[i+1] AS c2
CREATE (c1)-[:SIGUE_A]->(c2)
""", {"lid": libro_id})
def vectorize_chunks(libro_id: str):
"""Generate embeddings for chunks that don't have them yet."""
try:
from google import genai
client = genai.Client(api_key=os.getenv("GCP_API_KEY", ""))
except Exception as e:
print(f"GenAI init error: {e}")
return 0
chunks = query("""
MATCH (c:Chunk {libro_id: $lid})
WHERE c.embedding IS NULL
RETURN c.id AS id, c.text AS text,
c.titulo_capitulo AS titulo_capitulo,
c.titulo_seccion AS titulo_seccion,
c.tipo_contenido AS tipo_contenido
ORDER BY c.chunk_index
""", {"lid": libro_id})
if not chunks:
return 0
total = 0
embed_batch = 20
for i in range(0, len(chunks), embed_batch):
batch = chunks[i:i + embed_batch]
# Build contextual text for embedding
texts = []
for c in batch:
parts = []
if c.get("titulo_capitulo"):
parts.append(f"Capitulo: {c['titulo_capitulo']}")
if c.get("titulo_seccion"):
parts.append(f"Seccion: {c['titulo_seccion']}")
prefix = ". ".join(parts)
text = f"{prefix}. {c['text']}" if prefix else c["text"]
texts.append(text[:2000])
try:
result = client.models.embed_content(
model="gemini-embedding-2-preview",
contents=texts,
)
updates = [{"id": c["id"], "embedding": e.values}
for c, e in zip(batch, result.embeddings)]
write("""
UNWIND $updates AS u
MATCH (c:Chunk {id: u.id})
SET c.embedding = u.embedding
""", {"updates": updates})
total += len(batch)
except Exception as e:
print(f"Embedding error batch {i}: {e}")
time.sleep(5)
if i + embed_batch < len(chunks):
time.sleep(1)
return total
def run_ingest(libro_id: str, pdf_path: str, titulo: str, autor: str):
"""Full ingest pipeline. Updates _jobs dict with progress."""
job = {"status": "running", "progress": [], "result": None}
_jobs[libro_id] = job
def report(step, pct, msg):
job["progress"].append({"step": step, "pct": pct, "msg": msg})
try:
# Parse
report("parse", 0, f"Parseando {titulo}...")
children, parents = parse_and_chunk(pdf_path, libro_id)
report("parse", 100, f"Parseado: {len(children)} chunks, {len(parents)} parents")
# Upload
report("upload", 0, "Subiendo a Neo4j...")
upload_to_neo4j(libro_id, children, parents)
report("upload", 100, f"Subido: {len(children)} chunks a Neo4j")
# Vectorize
report("vectorize", 0, "Generando embeddings...")
n_emb = vectorize_chunks(libro_id)
report("vectorize", 100, f"Vectorizado: {n_emb} embeddings")
job["status"] = "completed"
job["result"] = {
"libro_id": libro_id,
"children": len(children),
"parents": len(parents),
"embeddings": n_emb,
}
except Exception as e:
job["status"] = "error"
job["result"] = {"error": str(e)}
finally:
# Cleanup temp file
if os.path.exists(pdf_path):
os.remove(pdf_path)

302
api/services/layers.py Normal file
View File

@ -0,0 +1,302 @@
"""Layer executors — cada capa busca en una dimensión del conocimiento.
Se ejecutan en paralelo via asyncio.gather.
"""
from services import graph, vector
def execute_ontology(analysis: dict) -> dict:
"""Traversal ontológico ATC + SNOMED. Bidireccional: sube y baja por ES_UN."""
result = {"farmacos_atc": [], "entidades_snomed": [], "query_cruzada": [], "categorias_encontradas": []}
for ent in analysis.get("entidades_detectadas", []):
buscar_en = ent.get("buscar_en", "")
texto = ent.get("texto", "")
if buscar_en == "CategoriaATC":
# Buscar categoría → bajar a fármacos (HACIA ABAJO)
farmacos = graph.query("""
MATCH (cat:CategoriaATC)
WHERE toLower(cat.nombre) CONTAINS toLower($term)
OPTIONAL MATCH (f:Farmaco)-[:ES_UN*1..5]->(cat)
RETURN cat.nombre AS categoria, cat.codigo AS codigo, cat.nivel AS nivel,
collect(DISTINCT f.nombre)[0..20] AS farmacos
""", {"term": texto})
for r in farmacos:
result["categorias_encontradas"].append({
"nombre": r["categoria"], "codigo": r["codigo"],
"nivel": r["nivel"], "tipo": "ATC",
"miembros": [f for f in (r["farmacos"] or []) if f]
})
elif buscar_en == "Farmaco":
# Buscar fármaco → subir a categorías (HACIA ARRIBA)
cats = graph.query("""
MATCH (f:Farmaco)-[:ES_UN*1..5]->(cat:CategoriaATC)
WHERE toLower(f.nombre) CONTAINS toLower($term)
RETURN f.nombre AS farmaco,
collect(DISTINCT {codigo: cat.codigo, nombre: cat.nombre, nivel: cat.nivel}) AS jerarquia
LIMIT 5
""", {"term": texto})
for r in cats:
result["farmacos_atc"].append({
"nombre": r["farmaco"],
"jerarquia": sorted(r["jerarquia"], key=lambda x: x.get("nivel", 0))
})
elif buscar_en == "CategoriaSNOMED":
# Buscar sistema/categoría → bajar a entidades
entidades = graph.query("""
MATCH (cat:CategoriaSNOMED)
WHERE toLower(cat.nombre) CONTAINS toLower($term)
OPTIONAL MATCH (e)-[:ES_UN*1..3]->(cat)
WHERE e:Patologia OR e:EstructuraAnatomica OR e:Procedimiento
RETURN cat.nombre AS categoria, cat.codigo AS codigo, cat.sistema AS sistema,
collect(DISTINCT {nombre: e.nombre, tipo: labels(e)[0]})[0..20] AS miembros
""", {"term": texto})
for r in entidades:
result["categorias_encontradas"].append({
"nombre": r["categoria"], "codigo": r["codigo"],
"sistema": r["sistema"], "tipo": "SNOMED",
"miembros": [m for m in (r["miembros"] or []) if m.get("nombre")]
})
elif buscar_en in ("Patologia", "EstructuraAnatomica", "Procedimiento", "Signo", "Sintoma", "MetodoDx", "Hallazgo", "Agente", "Parametro"):
# Buscar entidad → subir a categoría SNOMED
cats = graph.query(f"""
MATCH (e:{buscar_en})-[:ES_UN*1..3]->(cat:CategoriaSNOMED)
WHERE toLower(e.nombre) CONTAINS toLower($term)
RETURN e.nombre AS entidad, labels(e)[0] AS tipo,
collect(DISTINCT {{nombre: cat.nombre, sistema: cat.sistema, nivel: cat.nivel}}) AS jerarquia
LIMIT 5
""", {"term": texto})
for r in cats:
result["entidades_snomed"].append({
"nombre": r["entidad"], "tipo": r["tipo"],
"jerarquia": sorted(r["jerarquia"], key=lambda x: x.get("nivel", 0))
})
# Query cruzada: si hay ATC + SNOMED, cruzar
atc_cats = [c for c in result["categorias_encontradas"] if c["tipo"] == "ATC"]
snomed_cats = [c for c in result["categorias_encontradas"] if c["tipo"] == "SNOMED"]
if atc_cats and snomed_cats:
for atc in atc_cats[:2]:
for sno in snomed_cats[:2]:
cruzada = graph.query("""
MATCH (f:Farmaco)-[:ES_UN*1..5]->(atc:CategoriaATC {codigo: $atc_code})
MATCH (p:Patologia)-[:SE_TRATA_CON]->(f)
MATCH (p)-[:ES_UN*1..3]->(sno:CategoriaSNOMED {codigo: $sno_code})
RETURN DISTINCT f.nombre AS farmaco, p.nombre AS patologia
LIMIT 15
""", {"atc_code": atc["codigo"], "sno_code": sno["codigo"]})
for r in cruzada:
result["query_cruzada"].append({
"farmaco": r["farmaco"], "patologia": r["patologia"],
"clase_atc": atc["nombre"], "sistema_snomed": sno.get("sistema", "")
})
return result
def execute_graph(analysis: dict) -> dict:
"""Busca relaciones directas en el grafo para entidades detectadas."""
result = {"patologias": [], "procedimientos": [], "relaciones": []}
for ent in analysis.get("entidades_detectadas", []):
buscar_en = ent.get("buscar_en", "")
texto = ent.get("texto", "")
if buscar_en == "Patologia":
patos = graph.get_pathology(texto)
if patos:
result["patologias"].extend(patos[:5])
elif buscar_en == "Procedimiento":
procs = graph.get_procedure(texto)
if procs:
result["procedimientos"].extend(procs[:3])
# Buscar relaciones genéricas para cualquier entidad
if buscar_en in ("Patologia", "Farmaco", "EstructuraAnatomica", "Procedimiento",
"Signo", "Sintoma", "MetodoDx", "Agente"):
try:
rels = graph.query(f"""
MATCH (e:{buscar_en})-[r]-(related)
WHERE toLower(e.nombre) CONTAINS toLower($term)
AND NOT related:Chunk AND NOT related:ParentChunk
RETURN e.nombre AS desde, type(r) AS relacion, related.nombre AS hasta,
labels(related)[0] AS tipo_hasta
LIMIT 15
""", {"term": texto})
for r in rels:
result["relaciones"].append({
"desde": r["desde"], "relacion": r["relacion"],
"hasta": r["hasta"], "tipo_hasta": r["tipo_hasta"]
})
except Exception:
pass
return result
def execute_bibliography(analysis: dict, top_k: int = 20) -> dict:
"""Búsqueda híbrida multi-query: cada sub-query busca un aspecto del tema."""
import logging
sub_queries = analysis.get("sub_queries", [analysis["original"]])
if not sub_queries:
sub_queries = [analysis["original"]]
# Limitar a 5 sub-queries max para no explotar latencia
sub_queries = sub_queries[:5]
all_results = {} # id -> chunk (dedup)
total_keyword = 0
total_semantic = 0
for sq in sub_queries:
try:
# Cada sub-query busca un pool amplio para maximizar recall
per_query_k = max(20, top_k)
result = vector.search_hybrid(sq, top_k=per_query_k)
total_keyword += result.get("keyword_count", 0)
total_semantic += result.get("semantic_count", 0)
for chunk in result.get("results", []):
cid = chunk.get("id", "")
if cid not in all_results:
all_results[cid] = chunk
else:
# Si ya existe, sumar score (boost por aparecer en múltiples sub-queries)
all_results[cid]["rrf_score"] = all_results[cid].get("rrf_score", 0) + chunk.get("rrf_score", 0)
except Exception as e:
logging.error(f"Bibliography sub-query failed '{sq}': {type(e).__name__}: {str(e)[:80]}")
# Ordenar por score acumulado y tomar top_k
sorted_results = sorted(all_results.values(), key=lambda x: x.get("rrf_score", 0), reverse=True)
final = sorted_results[:top_k]
return {
"keyword_count": total_keyword,
"semantic_count": total_semantic,
"sub_queries_ejecutadas": len(sub_queries),
"chunks_unicos_encontrados": len(all_results),
"intencion": analysis.get("intencion", "general"),
"query_expandida": analysis.get("expandida", ""),
"results": final,
}
def execute_activities(analysis: dict) -> dict:
"""Busca actividades académicas (TPs, seminarios, talleres).
Siempre busca usando TODAS las entidades detectadas (no solo las
clasificadas como actividad_academica), porque el analyzer raramente
clasifica entidades como actividades.
"""
result = {"actividades": [], "material": []}
existing_ids = set()
def _add_activities(acts):
for act in (acts or [])[:5]:
aid = act.get("id")
if aid and aid not in existing_ids:
existing_ids.add(aid)
result["actividades"].append(act)
try:
mat = graph.get_activity_material(aid)
if mat:
result["material"].extend(mat)
except Exception:
pass
# 1. Buscar por cada entidad detectada (todas, no solo actividades)
for ent in analysis.get("entidades_detectadas", []):
texto = ent.get("texto", "")
if texto and len(texto) >= 3:
_add_activities(graph.get_activity(texto))
# 2. Buscar por sub_queries
for sq in analysis.get("sub_queries", []):
_add_activities(graph.get_activity(sq))
# 3. Buscar por palabras individuales del query original (fallback)
if not result["actividades"]:
pregunta = analysis.get("original", "")
words = [w for w in pregunta.lower().split() if len(w) >= 4]
for word in words[:6]:
_add_activities(graph.get_activity(word))
if result["actividades"]:
break
return result
def execute_dags(analysis: dict) -> dict:
"""Busca flujos clínicos (PATHWAY + CLINICAL) para entidades detectadas."""
result = {"pathways": [], "clinical": []}
search_terms = set()
for ent in analysis.get("entidades_detectadas", []):
search_terms.add(ent.get("texto", ""))
for term in search_terms:
if not term:
continue
# Pathways
try:
pathways = graph.query("""
MATCH (a)-[r:PATHWAY]->(b)
WHERE toLower(a.nombre) CONTAINS toLower($term)
OR toLower(b.nombre) CONTAINS toLower($term)
OR r.nombre_dag CONTAINS toLower($term)
RETURN DISTINCT r.nombre_dag AS dag, a.nombre AS desde,
b.nombre AS hasta, r.orden AS orden, r.nota AS nota
ORDER BY r.nombre_dag, r.orden
""", {"term": term})
dags = {}
for p in pathways:
dag_name = p["dag"]
if dag_name not in dags:
dags[dag_name] = {"nombre": dag_name, "pasos": []}
dags[dag_name]["pasos"].append({
"orden": p["orden"], "desde": p["desde"],
"hasta": p["hasta"], "nota": p.get("nota")
})
result["pathways"].extend(dags.values())
except Exception:
pass
# Clinical
try:
steps = graph.query("""
MATCH (a)-[r:CLINICAL]->(b)
WHERE toLower(a.nombre) CONTAINS toLower($term)
OR toLower(b.nombre) CONTAINS toLower($term)
OR r.nombre_dag CONTAINS toLower($term)
RETURN DISTINCT r.nombre_dag AS dag, a.nombre AS desde,
b.nombre AS hasta, r.orden AS orden,
r.tipo_paso AS tipo_paso, r.condicion AS condicion, r.nota AS nota
ORDER BY r.nombre_dag, r.orden, r.condicion
""", {"term": term})
dags = {}
for s in steps:
dag_name = s["dag"]
if dag_name not in dags:
dags[dag_name] = {"nombre": dag_name, "pasos": []}
dags[dag_name]["pasos"].append({
"orden": s["orden"], "desde": s["desde"],
"hasta": s["hasta"], "tipo_paso": s.get("tipo_paso"),
"condicion": s.get("condicion"), "nota": s.get("nota")
})
result["clinical"].extend(dags.values())
except Exception:
pass
return result

221
api/services/query.py Normal file
View File

@ -0,0 +1,221 @@
"""Query preprocessing: clasificación de intención, expansión de sinónimos y decomposición."""
import re
# Sinónimos médicos comunes (abreviatura → expansiones)
SINONIMOS = {
# Cardiología
"hta": ["hipertension arterial"],
"iam": ["infarto agudo de miocardio", "infarto"],
"icc": ["insuficiencia cardiaca congestiva", "insuficiencia cardiaca"],
"ecg": ["electrocardiograma"],
"fa": ["fibrilacion auricular"],
"tev": ["tromboembolismo venoso"],
"tep": ["tromboembolismo pulmonar", "embolia pulmonar"],
"tvp": ["trombosis venosa profunda"],
# Neumología
"epoc": ["enfermedad pulmonar obstructiva cronica"],
"sdra": ["sindrome de dificultad respiratoria aguda"],
"nac": ["neumonia adquirida en la comunidad"],
"rx": ["radiografia"],
# Endocrinología
"dbt": ["diabetes", "diabetes mellitus"],
"dm": ["diabetes mellitus"],
"dm2": ["diabetes mellitus tipo 2"],
"dm1": ["diabetes mellitus tipo 1"],
"tsh": ["tirotropina", "hormona estimulante de tiroides"],
# Infectología
"hiv": ["virus de inmunodeficiencia humana", "vih", "sida"],
"hbv": ["hepatitis b", "virus hepatitis b"],
"hcv": ["hepatitis c", "virus hepatitis c"],
"tbc": ["tuberculosis"],
"egb": ["estreptococo grupo b"],
"its": ["infeccion de transmision sexual"],
"itu": ["infeccion del tracto urinario", "infeccion urinaria"],
# Nefrología
"irc": ["insuficiencia renal cronica"],
"ira": ["insuficiencia renal aguda"],
"tfg": ["tasa de filtracion glomerular"],
# Gastroenterología
"eii": ["enfermedad inflamatoria intestinal"],
"rge": ["reflujo gastroesofagico"],
# Neurología
"acv": ["accidente cerebrovascular", "stroke"],
"ait": ["accidente isquemico transitorio"],
"lcr": ["liquido cefalorraquideo"],
# Pediatría / Neonatología
"rn": ["recien nacido"],
"rnpt": ["recien nacido pretermino"],
"rnt": ["recien nacido de termino"],
"bpn": ["bajo peso al nacer"],
"apgar": ["apgar"],
"ehrn": ["enfermedad hemorragica del recien nacido"],
# Oftalmología
"av": ["agudeza visual"],
"cv": ["campo visual"],
"pio": ["presion intraocular"],
"dpar": ["defecto pupilar aferente relativo"],
# ORL
"oma": ["otitis media aguda"],
"ome": ["otitis media con efusion", "otitis media secretora"],
"cae": ["conducto auditivo externo"],
# Dermatología
"da": ["dermatitis atopica", "eccema atopico"],
# Farmacología
"aine": ["antiinflamatorio no esteroideo", "antiinflamatorios no esteroideos"],
"atb": ["antibiotico", "antibioticos"],
"vo": ["via oral"],
"im": ["intramuscular"],
"iv": ["intravenoso", "endovenoso"],
"sc": ["subcutaneo"],
"ev": ["endovenoso", "intravenoso"],
# Sinónimos anatómicos
"papila optica": ["disco optico", "cabeza del nervio optico"],
"disco optico": ["papila optica", "cabeza del nervio optico"],
"mt": ["membrana timpanica", "timpano"],
"membrana timpanica": ["timpano"],
}
# Patrones para clasificar intención
INTENCION_PATTERNS = {
"tratamiento": [
r"tratamiento\b", r"terapia\b", r"como se trata",
r"farmaco", r"medicament", r"dosis", r"posologia",
r"primera linea", r"segunda linea", r"se trata con",
],
"diagnostico": [
r"diagnostico\b", r"como se diagnostica", r"metodo dx",
r"criterios", r"laboratorio", r"imagen", r"ecografia",
r"estudios complementarios", r"diferencial",
],
"anatomia": [
r"anatomia\b", r"histologia\b", r"estructura",
r"ubicacion", r"donde se encuentra", r"partes de",
r"capas", r"tunica", r"musculo", r"nervio",
],
"fisiologia": [
r"fisiologia\b", r"mecanismo", r"funcion de",
r"como funciona", r"transduccion", r"via",
],
"clinica": [
r"clinica\b", r"sintomas", r"signos", r"manifestacion",
r"cuadro clinico", r"presenta con", r"cursa con",
],
"etiologia": [
r"etiologia\b", r"causa\b", r"causas\b", r"agente",
r"patogenia", r"fisiopatologia", r"por que se produce",
],
"epidemiologia": [
r"epidemiologia\b", r"prevalencia", r"incidencia",
r"frecuencia", r"factores de riesgo",
],
"procedimiento": [
r"procedimiento\b", r"tecnica\b", r"como se hace",
r"como se realiza", r"pasos", r"lista de cotejo",
],
}
def expandir_sinonimos(query_text: str) -> str:
"""Expande abreviaturas y sinónimos médicos en la query."""
words = query_text.lower().split()
expanded = list(words)
for i, word in enumerate(words):
clean = word.strip(".,;:?!()")
if clean in SINONIMOS:
# Agregar sinónimos al final
for sin in SINONIMOS[clean]:
expanded.append(sin)
# También buscar frases de 2 palabras
text_lower = query_text.lower()
for phrase, sins in SINONIMOS.items():
if " " in phrase and phrase in text_lower:
for sin in sins:
expanded.append(sin)
return " ".join(expanded)
def clasificar_intencion(query_text: str) -> str:
"""Clasifica la intención de la query médica."""
text_lower = query_text.lower()
scores = {}
for intencion, patterns in INTENCION_PATTERNS.items():
score = 0
for pattern in patterns:
if re.search(pattern, text_lower):
score += 1
if score > 0:
scores[intencion] = score
if not scores:
return "general"
return max(scores, key=scores.get)
def descomponer_query(query_text: str, intencion: str) -> list[str]:
"""Descompone una query compleja en sub-queries más específicas."""
# Para queries simples (1-3 palabras), no descomponer
words = query_text.strip().split()
if len(words) <= 3:
return [query_text]
# Para queries de diagnóstico diferencial
if "diferencial" in query_text.lower() or " vs " in query_text.lower():
# Extraer los dos términos
parts = re.split(r"\bvs\b|\bdiferencial\b|\bentre\b", query_text.lower())
parts = [p.strip() for p in parts if p.strip()]
if len(parts) >= 2:
return [f"{parts[0]} definicion clinica", f"{parts[1]} definicion clinica",
f"{parts[0]} {parts[1]} diferencias"]
return [query_text]
# Para queries largas (>6 palabras), mantener original + versión condensada
if len(words) > 6:
# Mantener query original + versión con solo sustantivos médicos (quitar preposiciones)
stop_words = {"de", "del", "la", "el", "los", "las", "un", "una", "y", "o", "en", "con", "por", "para", "que", "como", "se"}
filtered = [w for w in words if w.lower() not in stop_words]
if len(filtered) >= 2:
return [query_text, " ".join(filtered)]
return [query_text]
def preprocess(query_text: str) -> dict:
"""Pipeline completo de preprocesamiento de query.
Returns:
{
"original": "tratamiento de HTA",
"intencion": "tratamiento",
"expandida": "tratamiento de HTA hipertension arterial",
"sub_queries": ["tratamiento de HTA hipertension arterial"],
}
"""
intencion = clasificar_intencion(query_text)
expandida = expandir_sinonimos(query_text)
sub_queries = descomponer_query(expandida, intencion)
return {
"original": query_text,
"intencion": intencion,
"expandida": expandida,
"sub_queries": sub_queries,
}

200
api/services/vector.py Normal file
View File

@ -0,0 +1,200 @@
"""Servicio de busqueda: semantica, full-text y hybrid con RRF."""
import os
from dotenv import load_dotenv
from services.graph import query
load_dotenv()
_client = None
# Pool size para retrieval inicial (se fusionan despues)
RETRIEVAL_POOL = 60
# Constante k para Reciprocal Rank Fusion
RRF_K = 60
# Embedding config
EMBEDDING_MODEL = "gemini-embedding-2-preview"
GCP_API_KEY = os.getenv("GCP_API_KEY", "")
def get_embedding_client():
global _client
if _client is None:
from google import genai
_client = genai.Client(api_key=GCP_API_KEY)
return _client
def generate_embedding(text: str) -> list:
client = get_embedding_client()
truncated = text[:2000] if len(text) > 2000 else text
r = client.models.embed_content(
model=EMBEDDING_MODEL,
contents=truncated,
)
return r.embeddings[0].values
def search_semantic(query_text: str, top_k: int = 40, libro_id: str = None) -> list:
"""Búsqueda semántica con vector KNN sobre embeddings."""
embedding = generate_embedding(query_text)
if libro_id:
results = query("""
CALL db.index.vector.queryNodes('chunk_embeddings', $top_k, $embedding)
YIELD node AS c, score
WHERE c.libro_id = $libro_id
RETURN c.id AS id, c.libro_id AS libro, c.page_start AS pag_inicio,
c.page_end AS pag_fin, c.text AS texto, c.word_count AS palabras,
c.titulo_capitulo AS capitulo, c.titulo_seccion AS seccion,
c.tipo_contenido AS tipo, c.parent_id AS parent_id, score
ORDER BY score DESC
""", {"embedding": embedding, "top_k": top_k, "libro_id": libro_id})
else:
results = query("""
CALL db.index.vector.queryNodes('chunk_embeddings', $top_k, $embedding)
YIELD node AS c, score
RETURN c.id AS id, c.libro_id AS libro, c.page_start AS pag_inicio,
c.page_end AS pag_fin, c.text AS texto, c.word_count AS palabras,
c.titulo_capitulo AS capitulo, c.titulo_seccion AS seccion,
c.tipo_contenido AS tipo, c.parent_id AS parent_id, score
ORDER BY score DESC
""", {"embedding": embedding, "top_k": top_k})
return results
def search_keyword(query_text: str, top_k: int = 40, libro_id: str = None) -> list:
"""Búsqueda full-text con scoring BM25 sobre Chunk.text (Lucene via Neo4j)."""
if not query_text.strip():
return []
# Lucene query: OR entre términos para mayor recall,
# el scoring BM25 se encarga de rankear mejor los que tienen más matches
terms = query_text.strip().split()
lucene_query = " ".join(terms) # OR implícito en Lucene
params = {"query": lucene_query, "top_k": top_k}
if libro_id:
results = query("""
CALL db.index.fulltext.queryNodes('busqueda_chunks', $query)
YIELD node AS c, score
WHERE c.libro_id = $libro_id
RETURN c.id AS id, c.libro_id AS libro, c.page_start AS pag_inicio,
c.page_end AS pag_fin, c.text AS texto, c.word_count AS palabras,
c.titulo_capitulo AS capitulo, c.titulo_seccion AS seccion,
c.tipo_contenido AS tipo, c.parent_id AS parent_id, score
ORDER BY score DESC
LIMIT $top_k
""", params)
else:
results = query("""
CALL db.index.fulltext.queryNodes('busqueda_chunks', $query)
YIELD node AS c, score
RETURN c.id AS id, c.libro_id AS libro, c.page_start AS pag_inicio,
c.page_end AS pag_fin, c.text AS texto, c.word_count AS palabras,
c.titulo_capitulo AS capitulo, c.titulo_seccion AS seccion,
c.tipo_contenido AS tipo, c.parent_id AS parent_id, score
ORDER BY score DESC
LIMIT $top_k
""", params)
return results
def _rrf_fusion(rankings: list[list], k: int = RRF_K) -> list:
"""Reciprocal Rank Fusion: combina múltiples rankings en uno solo.
Para cada resultado en cada ranking, calcula score = 1/(k + rank).
Suma los scores de todos los rankings donde aparece cada chunk.
Chunks que aparecen en múltiples rankings suben al top.
"""
scores = {}
chunk_data = {}
for ranking in rankings:
for rank, result in enumerate(ranking):
chunk_id = result["id"]
rrf_score = 1.0 / (k + rank + 1)
if chunk_id not in scores:
scores[chunk_id] = 0.0
chunk_data[chunk_id] = result
scores[chunk_id] += rrf_score
# Ordenar por score RRF combinado
sorted_ids = sorted(scores.keys(), key=lambda x: scores[x], reverse=True)
results = []
for chunk_id in sorted_ids:
result = chunk_data[chunk_id]
result["rrf_score"] = round(scores[chunk_id], 6)
results.append(result)
return results
def search_hybrid(query_text: str, top_k: int = 8, libro_id: str = None) -> dict:
"""Búsqueda híbrida con query rewriting + full-text + semántica + RRF.
1. Query preprocessing (expansión sinónimos, clasificación intención, decomposición)
2. Para cada sub-query: full-text top POOL + dense top POOL
3. RRF fusion de todos los rankings
4. Devuelve top_k resultados finales
"""
from services.query import preprocess
# 1. Preprocess query
processed = preprocess(query_text)
sub_queries = processed["sub_queries"]
all_rankings = []
total_keyword = 0
total_semantic = 0
# 2. Multi-retrieval por sub-query
for sq in sub_queries:
# Full-text
try:
kw = search_keyword(sq, top_k=RETRIEVAL_POOL, libro_id=libro_id)
if kw:
all_rankings.append(kw)
total_keyword += len(kw)
except Exception as e:
import logging
logging.error(f"Keyword search failed for '{sq}': {type(e).__name__}: {str(e)[:100]}")
# Semantic (solo para la primera sub-query para no hacer muchas llamadas al embedding API)
if sq == sub_queries[0]:
try:
sem = search_semantic(processed["expandida"], top_k=RETRIEVAL_POOL, libro_id=libro_id)
if sem:
all_rankings.append(sem)
total_semantic += len(sem)
except Exception as e:
import logging
logging.error(f"Semantic search failed: {type(e).__name__}: {str(e)[:100]}")
if not all_rankings:
return {
"keyword_count": 0,
"semantic_count": 0,
"intencion": processed["intencion"],
"query_expandida": processed["expandida"],
"results": [],
}
# 3. RRF fusion de todos los rankings
fused = _rrf_fusion(all_rankings)
# 4. Top-k final
results = fused[:top_k]
return {
"keyword_count": total_keyword,
"semantic_count": total_semantic,
"intencion": processed["intencion"],
"query_expandida": processed["expandida"],
"results": results,
}

82
catalog.json Normal file
View File

@ -0,0 +1,82 @@
{
"libros": [
{
"id": "farreras-2020",
"archivo": "Farreras Rozman Medicina Interna 20a Edicion.pdf",
"titulo": "Medicina Interna",
"autor": "Farreras Valentí, P.; Rozman, C.",
"edicion": "20°",
"paginas": 2953,
"tipo_pdf": "digital",
"estado": "parseado",
"fecha_parseo": "2026-03-16T11:19:07.777125",
"chunks_generados": 3008
},
{
"id": "garcia-feijoo-2012",
"archivo": "Manual de Oftalmologia Garcia Feijoo (2012).pdf",
"titulo": "Manual de Oftalmología",
"autor": "García-Feijóo, J. y Pablo-Júlvez, L.E.",
"edicion": "1°",
"paginas": 377,
"tipo_pdf": "digital",
"estado": "parseado",
"fecha_parseo": "2026-03-16T11:19:10.353875",
"chunks_generados": 307,
"chunks_v2": 887,
"parents_v2": 296,
"fecha_parseo_v2": "2026-03-17T19:17:21.710814"
},
{
"id": "cie10",
"archivo": "CIE 10.pdf",
"titulo": "Clasificación Internacional de Enfermedades (CIE-10)",
"autor": "OMS",
"paginas": 303,
"tipo_pdf": "digital",
"estado": "parseado",
"fecha_parseo": "2026-03-16T11:19:11.552750",
"chunks_generados": 131,
"chunks_v2": 448,
"parents_v2": 150,
"fecha_parseo_v2": "2026-03-17T19:17:22.836141"
},
{
"id": "diamante-orl",
"archivo": "Diamante.pdf",
"titulo": "Otorrinolaringología y afecciones conexas",
"autor": "Diamante, Vicente",
"paginas": 323,
"tipo_pdf": "digital",
"estado": "parseado",
"fecha_parseo": "2026-03-16T12:00:22.405044",
"chunks_generados": 187,
"chunks_v2": 500,
"parents_v2": 167,
"fecha_parseo_v2": "2026-03-17T19:17:23.482406"
},
{
"id": "derma-clinica",
"archivo": "Fundamentos en dermatologia clinica.pdf",
"titulo": "Fundamentos en dermatología clínica",
"autor": "Varios",
"paginas": 157,
"tipo_pdf": "escaneado",
"estado": "parseado",
"fecha_parseo": "2026-03-17T17:54:08.911377",
"chunks_generados": 0,
"nota": "Parseado con OCR (Google Cloud Vision). PDF escaneado, no tiene texto extraíble. Requiere OCR (Tesseract o similar)"
}
],
"course_materials": {
"estado": "parseadas_parcialmente",
"nota": "Example catalog entry. Configure your own books."
},
"libros_faltantes": [
"Dubín - Electrocardiografía práctica",
"Harrison - Principios de Medicina Interna",
"Fernández Bussy - Enfermedades de la piel (UNR Editora)",
"San Román - Manual de diagnóstico por imágenes",
"Sanguinetti - Semiología, Semiotecnia y Medicina Interna"
]
}

57
dags/aines-clinical.yaml Normal file
View File

@ -0,0 +1,57 @@
nombre: AINEs — Selección farmacológica
tipo: clinical
descripcion: Flujo para elegir el AINE adecuado según contexto clínico
pasos:
- orden: 1
nodo: dolor agudo
tipo_nodo: sintoma
descripcion: "Evaluar intensidad (EVA), localización, duración, causa"
- orden: 2
nodo: ibuprofeno
tipo_nodo: farmaco
condicion: "Dolor leve-moderado, sin riesgo GI ni CV"
descripcion: "400-600 mg c/6-8h. Máx 2400 mg/día. Primera elección por perfil riesgo/beneficio"
- orden: 3
nodo: paracetamol
tipo_nodo: farmaco
condicion: "Dolor leve, fiebre, riesgo GI alto, embarazo"
descripcion: "500-1000 mg c/6-8h. Máx 3g/día. No antiinflamatorio. Hepatotóxico >4g/día. Antídoto: N-acetilcisteína"
- orden: 4
nodo: diclofenac
tipo_nodo: farmaco
condicion: "Dolor moderado-severo, componente inflamatorio importante"
descripcion: "50 mg c/8h o 75 mg c/12h. Máx 150 mg/día. Mayor riesgo CV que ibuprofeno"
- orden: 5
nodo: ketorolac
tipo_nodo: farmaco
condicion: "Dolor agudo severo, postoperatorio, cólico renal"
descripcion: "10 mg VO c/6h o 30 mg IM/IV. Máximo 5 días. Alto riesgo GI"
- orden: 6
nodo: naproxeno
tipo_nodo: farmaco
condicion: "Dolor crónico, inflamación sostenida, paciente con riesgo CV"
descripcion: "250-500 mg c/12h. Menor riesgo CV entre los AINEs. Buena opción crónica"
- orden: 7
nodo: celecoxib
tipo_nodo: farmaco
condicion: "Riesgo GI alto, necesita antiinflamatorio"
descripcion: "200 mg/día. COX-2 selectivo. Menor gastropatía. Contraindicado en riesgo CV alto"
- orden: 8
nodo: aspirina
tipo_nodo: farmaco
condicion: "Cardioprotección, fiebre reumática"
descripcion: "100 mg/día cardioprotección. 500-1000 mg analgésico. Contraindicado <16 años (Reye). Inhibe COX irreversiblemente"
- orden: 9
nodo: corticosteroides
tipo_nodo: grupo_farmacologico
condicion: "Inflamación severa, autoinmune, obstrucción de vía aérea, no responde a AINEs"
descripcion: "Prednisona 0.5-1 mg/kg/día. Dexametasona en crup/edema. Siempre curso corto si es posible. Efectos adversos: Cushing, osteoporosis, supresión HPA"

59
dags/infeccion-respiratoria-pathway.yaml Normal file
View File

@ -0,0 +1,59 @@
nombre: Infección respiratoria alta — Diagnóstico diferencial
tipo: pathway
descripcion: Flujo clínico para orientar diagnóstico ante síntomas respiratorios altos
pasos:
- orden: 1
nodo: odinofagia
tipo_nodo: sintoma
descripcion: "Dolor de garganta. Evaluar duración, intensidad, disfagia"
- orden: 2
nodo: fiebre
tipo_nodo: signo
descripcion: "Medir temperatura. <38°C orienta viral, >38.5°C considerar bacteriana"
- orden: 3
nodo: rinorrea
tipo_nodo: signo
descripcion: "Serosa = viral. Purulenta >10 días = sinusitis bacteriana"
- orden: 4
nodo: examen faríngeo
tipo_nodo: procedimiento
descripcion: "Evaluar: eritema, exudado, petequias palatinas, hipertrofia amigdalina"
- orden: 5
nodo: adenopatías cervicales
tipo_nodo: signo
descripcion: "Submaxilares dolorosas = faringitis EBHGA. Cervicales posteriores + esplenomegalia = mononucleosis"
- orden: 6
nodo: faringitis estreptocócica
tipo_nodo: patologia
condicion: "Exudado amigdalino + fiebre alta + adenopatía submaxilar + ausencia de tos"
descripcion: "Score de Centor ≥3. Test rápido o cultivo. Tratamiento: amoxicilina 50 mg/kg/día x 10 días"
- orden: 7
nodo: resfrío común
tipo_nodo: patologia
condicion: "Rinorrea + congestión + tos + febrícula. Sin exudado"
descripcion: "Autolimitado 5-7 días. Tratamiento sintomático. NO antibióticos"
- orden: 8
nodo: mononucleosis infecciosa
tipo_nodo: patologia
condicion: "Adolescente/adulto joven + faringitis + adenopatías generalizadas + esplenomegalia"
descripcion: "Solicitar hemograma (linfocitos atípicos), monotest, IgM anti-VCA. Reposo. NO amoxicilina (exantema)"
- orden: 9
nodo: sinusitis aguda
tipo_nodo: patologia
condicion: "Rinorrea purulenta >10 días O empeoramiento tras mejoría inicial"
descripcion: "Amoxicilina-clavulánico. Si alergia: moxifloxacino"
- orden: 10
nodo: laringitis
tipo_nodo: patologia
condicion: "Disfonía + tos perruna + estridor. Predomina en niños (crup)"
descripcion: "Viral. Dexametasona VO dosis única. Adrenalina nebulizada si estridor en reposo"

39
dags/linfoma-clinical.yaml Normal file
View File

@ -0,0 +1,39 @@
nombre: Linfoma — Manejo clínico
tipo: clinical
descripcion: Tratamiento según tipo de linfoma
pasos:
- orden: 1
nodo: linfoma de hodgkin
tipo_nodo: patologia
descripcion: "Esclerosis nodular (70%), celularidad mixta (20-25%), rico en linfocitos, depleción linfocitaria"
- orden: 2
nodo: ABVD
tipo_nodo: farmaco
descripcion: "Adriamicina + Bleomicina + Vinblastina + Dacarbazina. Estadios I-II: 2-4 ciclos + RT. Estadios III-IV: 6-8 ciclos"
- orden: 3
nodo: linfoma difuso de células grandes B
tipo_nodo: patologia
descripcion: "Linfoma más frecuente (35%). Agresivo. Subtipos: GCB vs ABC"
- orden: 4
nodo: R-CHOP
tipo_nodo: farmaco
descripcion: "Rituximab + Ciclofosfamida + Doxorrubicina + Vincristina + Prednisona. 6-8 ciclos c/21 días"
- orden: 5
nodo: linfoma folicular
tipo_nodo: patologia
descripcion: "Indolente. t(14;18)/BCL2. Curso crónico con recaídas"
- orden: 6
nodo: rituximab
tipo_nodo: farmaco
descripcion: "Anti-CD20. Mantenimiento cada 2 meses x 2 años post-inducción"
- orden: 7
nodo: linfoma de burkitt
tipo_nodo: patologia
descripcion: "Muy agresivo. t(8;14)/MYC. Patrón cielo estrellado. Quimio intensiva urgente"

44
dags/linfoma-pathway.yaml Normal file
View File

@ -0,0 +1,44 @@
nombre: Linfoma — Razonamiento diagnóstico
tipo: pathway
descripcion: Flujo clínico desde adenopatía hasta diagnóstico y estadificación de linfoma
pasos:
- orden: 1
nodo: adenopatía persistente
tipo_nodo: signo
descripcion: "Adenopatía >2 semanas, indolora, firme, no adherida"
- orden: 2
nodo: síntomas B
tipo_nodo: signo
descripcion: "Evaluar: fiebre >38°C, sudoración nocturna, pérdida de peso >10% en 6 meses"
- orden: 3
nodo: hemograma completo
tipo_nodo: metodo_dx
descripcion: "Leucocitosis con linfocitos atípicos, anemia, VSG elevada"
- orden: 4
nodo: biopsia ganglionar
tipo_nodo: procedimiento
descripcion: "Biopsia excisional de ganglio. NUNCA punción aspirativa. Preferir supraclavicular/laterocervical"
- orden: 5
nodo: inmunofenotipo
tipo_nodo: metodo_dx
descripcion: "CD20, CD3, CD5, CD10, CD23, ciclina D1, BCL2, BCL6, Ki-67"
- orden: 6
nodo: linfoma
tipo_nodo: patologia
descripcion: "Clasificar: Hodgkin vs No Hodgkin. Subtipo histológico según OMS"
- orden: 7
nodo: tomografía PET-TC
tipo_nodo: metodo_dx
descripcion: "Estadificación Ann Arbor (I-IV). Evaluar masa bulky (>10 cm)"
- orden: 8
nodo: biopsia de médula ósea
tipo_nodo: procedimiento
descripcion: "Completar estadificación. Infiltración medular = estadio IV"

94
dags/otitis-clinical.yaml Normal file
View File

@ -0,0 +1,94 @@
nombre: manejo-otalgia
tipo: clinical
descripcion: "Razonamiento clinico ante un paciente con otalgia"
pasos:
# Evaluacion inicial
- nodo: "otalgia"
tipo_nodo: sintoma
tipo_paso: inicio
orden: 1
- nodo: "anamnesis"
tipo_nodo: procedimiento
tipo_paso: evaluacion
orden: 2
nota: "edad, duracion, fiebre, antecedentes ORL, trauma"
- nodo: "otoscopia"
tipo_nodo: procedimiento
tipo_paso: evaluacion
orden: 3
# Bifurcacion segun hallazgo
- nodo: "otitis media aguda"
tipo_nodo: patologia
tipo_paso: decision
orden: 4
condicion: "MT abombada, eritematosa, opaca"
- nodo: "otitis externa"
tipo_nodo: patologia
tipo_paso: decision
orden: 4
condicion: "CAE edematoso, signo del trago positivo, MT normal"
- nodo: "otitis media efusiva"
tipo_nodo: patologia
tipo_paso: decision
orden: 4
condicion: "MT retraida, nivel hidroaereo, sin signos inflamatorios"
- nodo: "otalgia referida"
tipo_nodo: patologia
tipo_paso: decision
orden: 4
condicion: "MT normal, CAE normal, otoscopia sin hallazgos"
# Acciones segun diagnostico
- nodo: "amoxicilina"
tipo_nodo: farmaco
tipo_paso: accion
orden: 5
condicion: "si OMA confirmada"
nota: "80-90 mg/kg/dia, 10 dias"
- nodo: "gotas oticas con antibiotico"
tipo_nodo: farmaco
tipo_paso: accion
orden: 5
condicion: "si otitis externa"
nota: "ciprofloxacina otica"
- nodo: "conducta expectante"
tipo_nodo: procedimiento
tipo_paso: accion
orden: 5
condicion: "si otitis media efusiva"
nota: "control en 3 meses"
- nodo: "evaluar ATM y faringe"
tipo_nodo: procedimiento
tipo_paso: accion
orden: 5
condicion: "si otalgia referida"
# Seguimiento
- nodo: "control 48-72hs"
tipo_nodo: procedimiento
tipo_paso: seguimiento
orden: 6
condicion: "si OMA con amoxicilina"
- nodo: "amoxicilina-clavulanico"
tipo_nodo: farmaco
tipo_paso: accion
orden: 7
condicion: "si no mejora en 48-72hs"
nota: "sospecha de resistencia"
- nodo: "derivacion ORL"
tipo_nodo: procedimiento
tipo_paso: accion
orden: 7
condicion: "si sospecha de mastoiditis o complicacion"

50
dags/otitis-pathway.yaml Normal file
View File

@ -0,0 +1,50 @@
nombre: fisiopatologia-oma
tipo: pathway
descripcion: "Cadena causal de la otitis media aguda: desde el agente hasta las complicaciones"
pasos:
- nodo: "streptococcus pneumoniae"
tipo_nodo: agente
orden: 1
- nodo: "colonizacion nasofaringea"
tipo_nodo: hallazgo
orden: 2
- nodo: "disfuncion tubaria"
tipo_nodo: hallazgo
orden: 3
- nodo: "infeccion oido medio"
tipo_nodo: patologia
orden: 4
- nodo: "inflamacion mucosa del oido medio"
tipo_nodo: hallazgo
orden: 5
- nodo: "acumulacion de exudado"
tipo_nodo: hallazgo
orden: 6
- nodo: "abombamiento de membrana timpanica"
tipo_nodo: signo
orden: 7
- nodo: "otalgia"
tipo_nodo: sintoma
orden: 8
- nodo: "hipoacusia conductiva"
tipo_nodo: hallazgo
orden: 9
- nodo: "perforacion timpanica"
tipo_nodo: signo
orden: 10
nota: "complicacion si no se trata"
- nodo: "mastoiditis"
tipo_nodo: patologia
orden: 11
nota: "complicacion grave"

47
db.py Normal file
View File

@ -0,0 +1,47 @@
"""Conexión a Neo4j y operaciones base."""
import os
from neo4j import GraphDatabase
from dotenv import load_dotenv
load_dotenv()
URI = os.getenv("NEO4J_URI")
USER = os.getenv("NEO4J_USERNAME")
PASSWORD = os.getenv("NEO4J_PASSWORD")
DATABASE = os.getenv("NEO4J_DATABASE")
def get_driver():
return GraphDatabase.driver(URI, auth=(USER, PASSWORD))
def run_query(query: str, params: dict = None):
"""Ejecuta una query Cypher y retorna los resultados."""
with get_driver() as driver:
with driver.session(database=DATABASE) as session:
result = session.run(query, params or {})
return [record.data() for record in result]
def run_write(query: str, params: dict = None):
"""Ejecuta una query de escritura."""
with get_driver() as driver:
with driver.session(database=DATABASE) as session:
session.execute_write(lambda tx: tx.run(query, params or {}))
def test_connection():
"""Verifica conexión a Neo4j."""
try:
with get_driver() as driver:
driver.verify_connectivity()
print("Conexión exitosa a Neo4j AuraDB")
return True
except Exception as e:
print(f"Error de conexión: {e}")
return False
if __name__ == "__main__":
test_connection()

307
dedup_entities.py Normal file
View File

@ -0,0 +1,307 @@
"""Deduplicación de entidades en Neo4j - Fase 1: Normalización de acentos.
Encuentra entidades duplicadas que difieren solo por acentos/tildes
y las mergea en un nodo canónico, transfiriendo todas las relaciones.
v2: Retry con backoff, batch Cypher por label, checkpoint para retomar.
CLI:
python dedup_entities.py # dry-run (solo reporta)
python dedup_entities.py --execute # ejecuta merge
python dedup_entities.py --label Patologia # solo un label
"""
import sys
import os
import json
import time
import unicodedata
# Fix Windows encoding
if sys.stdout.encoding != 'utf-8':
sys.stdout.reconfigure(encoding='utf-8')
from db import run_query, run_write
ENTITY_LABELS = [
"Patologia", "EstructuraAnatomica", "Procedimiento", "Farmaco",
"GrupoFarmacologico", "Agente", "Signo", "Sintoma",
"MetodoDx", "Hallazgo", "Parametro",
]
CHECKPOINT_FILE = os.path.join(os.path.dirname(__file__), "dedup_checkpoint.json")
MAX_RETRIES = 3
RETRY_DELAY = 2 # seconds, doubles each retry
def strip_accents(s: str) -> str:
nfkd = unicodedata.normalize('NFKD', s)
return ''.join(c for c in nfkd if not unicodedata.combining(c))
def has_accents(s: str) -> bool:
return s != strip_accents(s)
def retry_query(func, *args, **kwargs):
"""Execute a DB function with retry + exponential backoff."""
for attempt in range(MAX_RETRIES + 1):
try:
return func(*args, **kwargs)
except Exception as e:
if attempt == MAX_RETRIES:
raise
delay = RETRY_DELAY * (2 ** attempt)
print(f" Retry {attempt+1}/{MAX_RETRIES} in {delay}s: {str(e)[:60]}")
time.sleep(delay)
def pick_canonical(nodes: list) -> dict:
return max(nodes, key=lambda n: (
has_accents(n["nombre"]),
n.get("freq") or 0,
len(n.get("sinonimos") or []),
))
def load_checkpoint() -> dict:
if os.path.exists(CHECKPOINT_FILE):
with open(CHECKPOINT_FILE, "r") as f:
return json.load(f)
return {"completed_labels": [], "stats": {}}
def save_checkpoint(data: dict):
with open(CHECKPOINT_FILE, "w") as f:
json.dump(data, f, indent=2)
def fetch_entities(label: str) -> list:
return retry_query(run_query, f"""
MATCH (n:{label})
RETURN elementId(n) AS eid, n.nombre AS nombre,
n.freq AS freq, n.sinonimos AS sinonimos
""")
def find_accent_groups(entities: list) -> dict:
groups = {}
for ent in entities:
key = strip_accents(ent["nombre"].lower().strip())
groups.setdefault(key, []).append(ent)
return {k: v for k, v in groups.items() if len(v) > 1}
def merge_group_batch(canon: dict, duplicates: list, label: str) -> dict:
"""Mergea un grupo usando queries batch con retry."""
canon_eid = canon["eid"]
total = {"menciona": 0, "rels_out": 0, "rels_in": 0, "deleted": 0}
all_sinonimos = set(canon.get("sinonimos") or [])
total_freq = canon.get("freq") or 0
dup_eids = [d["eid"] for d in duplicates]
for dup in duplicates:
all_sinonimos.add(dup["nombre"])
for s in (dup.get("sinonimos") or []):
all_sinonimos.add(s)
total_freq += dup.get("freq") or 0
# 1. Batch: transferir MENCIONA de todos los dups al canon
menciona = retry_query(run_query, """
UNWIND $dup_eids AS deid
MATCH (c:Chunk)-[:MENCIONA]->(dup) WHERE elementId(dup) = deid
RETURN DISTINCT elementId(c) AS chunk_eid
""", {"dup_eids": dup_eids})
if menciona:
retry_query(run_write, """
UNWIND $chunks AS ceid
MATCH (c) WHERE elementId(c) = ceid
MATCH (canon) WHERE elementId(canon) = $canon_eid
MERGE (c)-[:MENCIONA]->(canon)
""", {"chunks": [m["chunk_eid"] for m in menciona], "canon_eid": canon_eid})
total["menciona"] = len(menciona)
# 2. Batch: transferir rels outgoing de todos los dups
rels_out = retry_query(run_query, """
UNWIND $dup_eids AS deid
MATCH (dup)-[r]->(target)
WHERE elementId(dup) = deid
AND elementId(target) <> $canon_eid
AND NOT elementId(target) IN $dup_eids
AND type(r) <> 'MENCIONA'
RETURN DISTINCT type(r) AS rtype, elementId(target) AS target_eid
""", {"dup_eids": dup_eids, "canon_eid": canon_eid})
if rels_out:
by_type = {}
for r in rels_out:
by_type.setdefault(r["rtype"], []).append(r["target_eid"])
for rtype, targets in by_type.items():
retry_query(run_write, f"""
UNWIND $targets AS teid
MATCH (canon) WHERE elementId(canon) = $canon_eid
MATCH (t) WHERE elementId(t) = teid
MERGE (canon)-[:{rtype}]->(t)
""", {"targets": targets, "canon_eid": canon_eid})
total["rels_out"] = len(rels_out)
# 3. Batch: transferir rels incoming de todos los dups
rels_in = retry_query(run_query, """
UNWIND $dup_eids AS deid
MATCH (source)-[r]->(dup)
WHERE elementId(dup) = deid
AND elementId(source) <> $canon_eid
AND NOT elementId(source) IN $dup_eids
AND type(r) <> 'MENCIONA'
RETURN DISTINCT type(r) AS rtype, elementId(source) AS source_eid
""", {"dup_eids": dup_eids, "canon_eid": canon_eid})
if rels_in:
by_type = {}
for r in rels_in:
by_type.setdefault(r["rtype"], []).append(r["source_eid"])
for rtype, sources in by_type.items():
retry_query(run_write, f"""
UNWIND $sources AS seid
MATCH (s) WHERE elementId(s) = seid
MATCH (canon) WHERE elementId(canon) = $canon_eid
MERGE (s)-[:{rtype}]->(canon)
""", {"sources": sources, "canon_eid": canon_eid})
total["rels_in"] = len(rels_in)
# 4. Batch: DETACH DELETE todos los dups de una vez
retry_query(run_write, """
UNWIND $dup_eids AS deid
MATCH (n) WHERE elementId(n) = deid
DETACH DELETE n
""", {"dup_eids": dup_eids})
total["deleted"] = len(duplicates)
# 5. Actualizar canónico
all_sinonimos.discard(canon["nombre"])
retry_query(run_write, """
MATCH (n) WHERE elementId(n) = $eid
SET n.sinonimos = $sins, n.freq = $freq
""", {"eid": canon_eid, "sins": sorted(list(all_sinonimos)), "freq": total_freq})
return total
def dedup_label(label: str, execute: bool = False) -> dict:
entities = fetch_entities(label)
groups = find_accent_groups(entities)
if not groups:
return {"groups": 0, "deleted": 0, "menciona": 0, "rels": 0}
total = {"groups": len(groups), "deleted": 0, "menciona": 0, "rels": 0}
processed = 0
for key, nodes in groups.items():
canon = pick_canonical(nodes)
dups = [n for n in nodes if n["eid"] != canon["eid"]]
if not execute:
dup_names = [d["nombre"] for d in dups]
print(f" {canon['nombre']} <- {dup_names}")
total["deleted"] += len(dups)
continue
try:
stats = merge_group_batch(canon, dups, label)
total["deleted"] += stats["deleted"]
total["menciona"] += stats["menciona"]
total["rels"] += stats["rels_out"] + stats["rels_in"]
processed += 1
if processed % 50 == 0:
print(f" [{processed}/{len(groups)}] {total['deleted']} eliminados")
except Exception as e:
print(f" ERROR en grupo '{key}': {str(e)[:80]}")
continue
return total
def main():
execute = "--execute" in sys.argv
target_label = None
if "--label" in sys.argv:
idx = sys.argv.index("--label")
target_label = sys.argv[idx + 1] if idx + 1 < len(sys.argv) else None
labels = [target_label] if target_label else ENTITY_LABELS
mode = "EXECUTE" if execute else "DRY-RUN"
# Load checkpoint
checkpoint = load_checkpoint()
if execute and not target_label:
completed = checkpoint.get("completed_labels", [])
remaining = [l for l in labels if l not in completed]
if completed:
print(f" Retomando desde checkpoint. Ya completados: {completed}")
labels = remaining
print(f"\n{'='*60}")
print(f" DEDUP FASE 1 v2: Normalización de acentos [{mode}]")
print(f" Labels a procesar: {labels}")
print(f"{'='*60}\n")
grand_total = {"groups": 0, "deleted": 0, "menciona": 0, "rels": 0}
for label in labels:
print(f"--- {label} ---")
t0 = time.time()
try:
stats = dedup_label(label, execute=execute)
except Exception as e:
print(f" FATAL en {label}: {str(e)[:100]}")
print(f" Guardando checkpoint y saliendo...")
save_checkpoint(checkpoint)
sys.exit(1)
elapsed = time.time() - t0
for k in grand_total:
grand_total[k] += stats[k]
if stats["groups"] == 0:
print(" Sin duplicados por acentos")
else:
print(f" {stats['groups']} grupos, {stats['deleted']} nodos eliminados" +
(f", {stats['menciona']} MENCIONA, {stats['rels']} rels ({elapsed:.1f}s)" if execute else ""))
# Save checkpoint per label
if execute:
checkpoint["completed_labels"] = checkpoint.get("completed_labels", []) + [label]
checkpoint["stats"] = checkpoint.get("stats", {})
checkpoint["stats"][label] = stats
save_checkpoint(checkpoint)
print(f" Checkpoint guardado.")
print()
# Resumen
print(f"{'='*60}")
print(f" RESUMEN {'(ejecutado)' if execute else '(dry-run)'}")
print(f"{'='*60}")
print(f" Grupos duplicados: {grand_total['groups']}")
print(f" Nodos {'eliminados' if execute else 'a eliminar'}: {grand_total['deleted']}")
if execute:
print(f" MENCIONA transferidas: {grand_total['menciona']}")
print(f" Relaciones transferidas: {grand_total['rels']}")
# Limpiar checkpoint al terminar
if os.path.exists(CHECKPOINT_FILE):
os.remove(CHECKPOINT_FILE)
print(" Checkpoint limpiado (todo completado).")
print()
if __name__ == "__main__":
main()

63
docker-compose.yml Normal file
View File

@ -0,0 +1,63 @@
# MedGraph — Docker Compose
#
# Runs Neo4j + MedGraph API locally.
#
# Usage:
# docker-compose up -d # start services
# docker-compose down # stop services
# docker-compose logs -f api # view API logs
#
# After starting, run: python quickstart.py
# to populate the database with your first book.
services:
# Neo4j Community Edition
# Web UI: http://localhost:7474
# Bolt: bolt://localhost:7687
neo4j:
image: neo4j:5-community
container_name: medgraph-neo4j
ports:
- "7474:7474" # HTTP (browser UI)
- "7687:7687" # Bolt (driver connection)
environment:
- NEO4J_AUTH=neo4j/changeme-local-password
- NEO4J_PLUGINS=["apoc"]
- NEO4J_dbms_security_procedures_unrestricted=apoc.*
volumes:
- neo4j_data:/data
- neo4j_logs:/logs
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:7474 || exit 1"]
interval: 10s
timeout: 5s
retries: 5
# MedGraph API (FastAPI)
# http://localhost:8000
# Docs: http://localhost:8000/docs
api:
build:
context: ./api
dockerfile: Dockerfile
container_name: medgraph-api
ports:
- "8000:8080"
environment:
- NEO4J_URI=bolt://neo4j:7687
- NEO4J_USERNAME=neo4j
- NEO4J_PASSWORD=changeme-local-password
- NEO4J_DATABASE=neo4j
- API_KEY=${API_KEY:-medgraph-local}
- GCP_API_KEY=${GCP_API_KEY:-}
- GCP_PROJECT=${GCP_PROJECT:-}
- ENVIRONMENT=development
depends_on:
neo4j:
condition: service_healthy
restart: unless-stopped
volumes:
neo4j_data:
neo4j_logs:

24
examples/README.md Normal file
View File

@ -0,0 +1,24 @@
# Examples
Place your PDFs here and run `quickstart.py` to process them through the full MedGraph pipeline.
## Getting a test PDF
MedGraph doesn't include copyrighted textbooks. You need to bring your own. For testing, you can use any freely available medical document:
- **WHO Guidelines**: [who.int/publications](https://www.who.int/publications)
- **PubMed Central**: [ncbi.nlm.nih.gov/pmc](https://www.ncbi.nlm.nih.gov/pmc/) (open access papers)
## How it works
1. Put a PDF in this folder
2. Run `python quickstart.py`
3. The pipeline will: parse → chunk → vectorize → extract entities
4. Everything is stored in YOUR Neo4j instance — nothing leaves your infrastructure
## Important
- No copyrighted material is included in this repository
- The PDF content is processed locally
- Embeddings are generated via Google Gemini API (requires GCP credentials)
- Extracted entities are stored in your Neo4j instance

671
extract_entities.py Normal file
View File

@ -0,0 +1,671 @@
"""Extractor masivo de entidades medicas con LLM (Gemini).
Procesa chunks de texto medico y extrae entidades + relaciones
para poblar el grafo semantico de Neo4j.
Modulo importable. Funciones principales:
- extract_from_chunk(model, chunk) -> dict
- extract_libro(libro_id, model_id, dev_mode) -> stats
- upload_entities(entities, dev_mode) -> stats
CLI: python extract_entities.py <libro_id> [--preview] [--dev] [--limit N]
"""
import json
import os
import re
import time
import unicodedata
from datetime import datetime
from dotenv import load_dotenv
from db import run_write, run_query
load_dotenv()
PARSED_DIR = os.path.join(os.path.dirname(__file__), "parsed")
OUTPUT_DIR = os.path.join(os.path.dirname(__file__), "extracted")
DEFAULT_MODEL = "gemini-2.0-flash"
GCP_API_KEY = os.getenv("GCP_API_KEY", "")
# Rate limiting
BATCH_SIZE = 5 # chunks por batch (para no saturar)
DELAY_BETWEEN = 1.0 # segundos entre batches
MAX_RETRIES = 2
# Tipos de entidad validos
ENTITY_TYPES = {
"patologia", "estructura_anatomica", "procedimiento", "farmaco",
"grupo_farmacologico", "agente", "signo", "sintoma",
"metodo_dx", "hallazgo", "parametro",
}
# Relaciones validas
RELATION_TYPES = {
"CAUSADA_POR", "SE_MANIFIESTA_CON", "SE_DIAGNOSTICA_CON",
"SE_TRATA_CON", "PARTE_DE", "EVALUA", "PERTENECE_A",
"PUEDE_PRODUCIR", "DIFERENCIAL_DE", "ASOCIADA_A",
"IRRIGA", "INERVA", "DRENA_EN", "SE_ORIGINA_EN",
"FACTOR_DE_RIESGO", "COMPLICACION_DE", "VARIANTE_DE",
}
# Mapa tipo -> label Neo4j
TYPE_TO_LABEL = {
"patologia": "Patologia",
"estructura_anatomica": "EstructuraAnatomica",
"procedimiento": "Procedimiento",
"farmaco": "Farmaco",
"grupo_farmacologico": "GrupoFarmacologico",
"agente": "Agente",
"signo": "Signo",
"sintoma": "Sintoma",
"metodo_dx": "MetodoDx",
"hallazgo": "Hallazgo",
"parametro": "Parametro",
}
EXTRACTION_PROMPT = """Eres un extractor de entidades medicas. Analiza el siguiente texto de un libro medico y extrae TODAS las entidades y relaciones medicas que encuentres.
REGLAS:
- Extrae SOLO entidades medicas concretas (no conceptos vagos como "tratamiento" sin especificar cual)
- Usa nombres canonicos en espanol (ej: "otitis media aguda", no "OMA" ni "acute otitis media")
- Incluye sinonimos y abreviaturas comunes
- Las relaciones deben conectar entidades que aparecen en el texto
- Si no hay entidades medicas relevantes, devuelve listas vacias
- NO inventes relaciones que no esten implicitas o explicitas en el texto
TIPOS DE ENTIDAD:
- patologia: enfermedades, sindromes, trastornos
- estructura_anatomica: organos, tejidos, estructuras
- procedimiento: tecnicas diagnosticas o terapeuticas
- farmaco: medicamentos especificos
- grupo_farmacologico: familias de farmacos
- agente: microorganismos, virus, parasitos
- signo: hallazgos objetivos del examen fisico
- sintoma: manifestaciones subjetivas del paciente
- metodo_dx: estudios complementarios, laboratorio
- hallazgo: resultados de estudios o examenes
- parametro: valores medibles (presion arterial, frecuencia, etc.)
TIPOS DE RELACION:
- CAUSADA_POR: patologia <- agente/causa
- SE_MANIFIESTA_CON: patologia -> signo/sintoma
- SE_DIAGNOSTICA_CON: patologia -> metodo_dx/procedimiento
- SE_TRATA_CON: patologia -> farmaco/procedimiento
- PARTE_DE: estructura -> estructura mayor
- EVALUA: procedimiento -> estructura/parametro
- PERTENECE_A: farmaco -> grupo_farmacologico
- PUEDE_PRODUCIR: farmaco -> signo/efecto adverso
- DIFERENCIAL_DE: patologia <-> patologia
- ASOCIADA_A: entidad <-> entidad (relacion general)
- FACTOR_DE_RIESGO: entidad -> patologia
- COMPLICACION_DE: patologia -> patologia
- VARIANTE_DE: patologia -> patologia
TEXTO:
\"\"\"
{texto}
\"\"\"
CONTEXTO: Libro: {libro}, Capitulo: {capitulo}, Seccion: {seccion}
Responde SOLO con JSON valido, sin markdown ni explicaciones:
{{"entidades": [{{"nombre": "...", "tipo": "...", "sinonimos": ["..."]}}], "relaciones": [{{"desde": "...", "relacion": "...", "hasta": "..."}}]}}"""
def normalize_name(name: str) -> str:
"""Normaliza nombre de entidad: lowercase, sin acentos extra, trim."""
if not name:
return ""
name = name.strip().lower()
# Quitar puntos finales, parentesis sueltos
name = re.sub(r'[\.;,]+$', '', name).strip()
# Colapsar espacios
name = re.sub(r'\s+', ' ', name)
return name
def init_model(model_id: str = DEFAULT_MODEL):
"""Inicializa cliente de Google GenAI con API key."""
from google import genai
client = genai.Client(api_key=GCP_API_KEY)
print(f" Modelo inicializado: {model_id}")
return {"client": client, "model_id": model_id}
def extract_from_chunk(model, chunk: dict, libro_titulo: str = "") -> dict:
"""Extrae entidades y relaciones de un chunk usando el LLM.
Args:
model: dict con client (genai.Client) y model_id
chunk: dict con text, titulo_capitulo, titulo_seccion, libro_id
libro_titulo: Titulo del libro para contexto
Returns:
dict con entidades y relaciones validadas
"""
prompt = EXTRACTION_PROMPT.format(
texto=chunk["text"][:3000], # Limitar largo
libro=libro_titulo or chunk.get("libro_id", ""),
capitulo=chunk.get("titulo_capitulo", ""),
seccion=chunk.get("titulo_seccion", ""),
)
client = model["client"]
model_id = model["model_id"]
for attempt in range(MAX_RETRIES + 1):
try:
response = client.models.generate_content(
model=model_id,
contents=prompt,
config={
"temperature": 0.1,
"max_output_tokens": 2048,
"response_mime_type": "application/json",
},
)
text = response.text.strip()
# Parsear JSON
data = json.loads(text)
# Validar y limpiar
return _validate_extraction(data, chunk)
except json.JSONDecodeError:
# Intentar extraer JSON de respuesta con markdown
match = re.search(r'\{.*\}', text, re.DOTALL)
if match:
try:
data = json.loads(match.group())
return _validate_extraction(data, chunk)
except json.JSONDecodeError:
pass
if attempt < MAX_RETRIES:
time.sleep(2)
continue
return {"entidades": [], "relaciones": [], "error": "json_parse_error"}
except Exception as e:
if attempt < MAX_RETRIES:
time.sleep(3)
continue
return {"entidades": [], "relaciones": [], "error": str(e)}
def _validate_extraction(data: dict, chunk: dict) -> dict:
"""Valida y limpia entidades/relaciones extraidas."""
valid_entities = []
entity_names = set()
for ent in data.get("entidades", []):
nombre = normalize_name(ent.get("nombre", ""))
tipo = ent.get("tipo", "").lower().strip()
# Validar
if not nombre or len(nombre) < 2 or len(nombre) > 100:
continue
if tipo not in ENTITY_TYPES:
continue
# Normalizar sinonimos
sinonimos = []
for s in ent.get("sinonimos", []):
sn = normalize_name(s)
if sn and sn != nombre and len(sn) >= 2:
sinonimos.append(sn)
valid_entities.append({
"nombre": nombre,
"tipo": tipo,
"sinonimos": sinonimos,
})
entity_names.add(nombre)
valid_relations = []
for rel in data.get("relaciones", []):
desde = normalize_name(rel.get("desde", ""))
hasta = normalize_name(rel.get("hasta", ""))
relacion = rel.get("relacion", "").upper().strip()
if not desde or not hasta or not relacion:
continue
if relacion not in RELATION_TYPES:
continue
# Al menos uno de los extremos debe ser una entidad extraida
if desde not in entity_names and hasta not in entity_names:
continue
valid_relations.append({
"desde": desde,
"relacion": relacion,
"hasta": hasta,
})
return {
"entidades": valid_entities,
"relaciones": valid_relations,
"chunk_id": chunk.get("id", ""),
"libro_id": chunk.get("libro_id", ""),
}
def extract_libro(libro_id: str, model_id: str = DEFAULT_MODEL,
limit: int = 0, skip_first: int = 0,
on_progress: callable = None) -> dict:
"""Extrae entidades de todos los chunks de un libro.
Args:
libro_id: ID del libro
model_id: Model ID de Gemini
limit: Si > 0, solo procesar N chunks (para testing)
skip_first: Saltear los primeros N chunks (prologo, indice)
on_progress: Callback(pct, msg)
Returns:
dict con estadisticas y path al archivo de salida
"""
# Cargar chunks v2
chunks_path = os.path.join(PARSED_DIR, f"{libro_id}_v2_chunks.json")
if not os.path.exists(chunks_path):
raise FileNotFoundError(f"No hay chunks v2 para {libro_id}")
with open(chunks_path, "r", encoding="utf-8") as f:
all_chunks = json.load(f)
# Filtrar chunks de contenido (skip prologo/indice/colaboradores)
chunks = [c for c in all_chunks if c.get("word_count", 0) >= 100]
if skip_first:
chunks = chunks[skip_first:]
if limit:
chunks = chunks[:limit]
print(f" Procesando {len(chunks)} chunks de {libro_id}")
# Inicializar modelo
model = init_model(model_id)
# Cargar titulo del libro
from parser_v2 import load_catalog
catalog = load_catalog()
libro_entry = next((l for l in catalog["libros"] if l["id"] == libro_id), None)
libro_titulo = libro_entry["titulo"] if libro_entry else libro_id
# Procesar chunks
all_extractions = []
total_entities = 0
total_relations = 0
errors = 0
start_time = time.time()
for i, chunk in enumerate(chunks):
extraction = extract_from_chunk(model, chunk, libro_titulo)
all_extractions.append(extraction)
n_ent = len(extraction["entidades"])
n_rel = len(extraction["relaciones"])
total_entities += n_ent
total_relations += n_rel
if extraction.get("error"):
errors += 1
# Progress
if (i + 1) % 10 == 0 or i == len(chunks) - 1:
pct = int((i + 1) / len(chunks) * 100)
elapsed = time.time() - start_time
rate = (i + 1) / elapsed if elapsed > 0 else 0
eta = (len(chunks) - i - 1) / rate if rate > 0 else 0
msg = f" [{pct}%] {i+1}/{len(chunks)} chunks | {total_entities} entidades | {total_relations} relaciones | ETA: {eta:.0f}s"
print(msg)
if on_progress:
on_progress(pct, msg)
# Rate limiting
if (i + 1) % BATCH_SIZE == 0:
time.sleep(DELAY_BETWEEN)
# Guardar resultado
os.makedirs(OUTPUT_DIR, exist_ok=True)
output_path = os.path.join(OUTPUT_DIR, f"{libro_id}_entities.json")
result = {
"libro_id": libro_id,
"model_id": model_id,
"fecha": datetime.now().isoformat(),
"chunks_procesados": len(chunks),
"total_entidades": total_entities,
"total_relaciones": total_relations,
"errors": errors,
"extractions": all_extractions,
}
with open(output_path, "w", encoding="utf-8") as f:
json.dump(result, f, ensure_ascii=False, indent=2)
print(f"\n Extraccion completa: {total_entities} entidades, {total_relations} relaciones")
print(f" Errores: {errors}")
print(f" Guardado en: {output_path}")
return {
"total_entities": total_entities,
"total_relations": total_relations,
"errors": errors,
"output_path": output_path,
"duration": time.time() - start_time,
}
def canonicalize_entities(extractions: list) -> tuple:
"""Consolida entidades de multiples chunks en un set canonico.
Resuelve sinonimos, agrupa duplicados, cuenta frecuencias.
Returns:
(canonical_entities, canonical_relations)
"""
# Mapa nombre -> entidad canonica
entity_map = {} # nombre normalizado -> {nombre, tipo, sinonimos, freq, chunk_ids}
synonym_map = {} # sinonimo -> nombre canonico
for ext in extractions:
chunk_id = ext.get("chunk_id", "")
for ent in ext.get("entidades", []):
nombre = ent["nombre"]
tipo = ent["tipo"]
# Verificar si ya existe como sinonimo
canon_name = synonym_map.get(nombre, nombre)
if canon_name in entity_map:
# Ya existe: incrementar frecuencia, agregar chunk
existing = entity_map[canon_name]
existing["freq"] += 1
existing["chunk_ids"].add(chunk_id)
# Agregar sinonimos nuevos
for s in ent.get("sinonimos", []):
if s not in existing["sinonimos"] and s != canon_name:
existing["sinonimos"].append(s)
synonym_map[s] = canon_name
else:
# Nueva entidad
entity_map[nombre] = {
"nombre": nombre,
"tipo": tipo,
"sinonimos": list(ent.get("sinonimos", [])),
"freq": 1,
"chunk_ids": {chunk_id},
}
# Registrar sinonimos
for s in ent.get("sinonimos", []):
synonym_map[s] = nombre
# Consolidar relaciones
relation_set = set() # (desde, relacion, hasta)
canonical_relations = []
for ext in extractions:
for rel in ext.get("relaciones", []):
desde = synonym_map.get(rel["desde"], rel["desde"])
hasta = synonym_map.get(rel["hasta"], rel["hasta"])
key = (desde, rel["relacion"], hasta)
if key not in relation_set:
relation_set.add(key)
canonical_relations.append({
"desde": desde,
"relacion": rel["relacion"],
"hasta": hasta,
})
# Convertir chunk_ids set a list para JSON
canonical_entities = []
for ent in entity_map.values():
ent["chunk_ids"] = list(ent["chunk_ids"])
canonical_entities.append(ent)
# Ordenar por frecuencia
canonical_entities.sort(key=lambda x: x["freq"], reverse=True)
return canonical_entities, canonical_relations
def upload_entities(entities: list, relations: list, libro_id: str,
dev_mode: bool = True) -> dict:
"""Sube entidades y relaciones a Neo4j en batch (UNWIND).
Args:
entities: Lista de entidades canonicas
relations: Lista de relaciones canonicas
libro_id: ID del libro fuente
dev_mode: Si True, usa labels con sufijo Dev (PatologiaDev, etc.)
Returns:
dict con estadisticas
"""
suffix = "Dev" if dev_mode else ""
created_entities = 0
created_relations = 0
BATCH = 200
print(f" Subiendo {len(entities)} entidades{' (DEV mode)' if dev_mode else ''}...")
# Agrupar entidades por tipo (cada label necesita su propia query UNWIND)
by_type = {}
for ent in entities:
tipo = ent["tipo"]
if tipo not in by_type:
by_type[tipo] = []
by_type[tipo].append(ent)
# Subir entidades en batch por tipo
for tipo, ents in by_type.items():
label = TYPE_TO_LABEL.get(tipo, "Entidad") + suffix
for i in range(0, len(ents), BATCH):
batch = ents[i:i + BATCH]
batch_data = [{
"nombre": e["nombre"],
"tipo": e["tipo"],
"sinonimos": e.get("sinonimos", []),
"freq": e.get("freq", 1),
"libro_id": libro_id,
} for e in batch]
try:
run_write(f"""
UNWIND $batch AS ent
MERGE (e:{label} {{nombre: ent.nombre}})
SET e.tipo = ent.tipo,
e.sinonimos = ent.sinonimos,
e.freq = ent.freq,
e.fuente_libro = ent.libro_id
""", {"batch": batch_data})
created_entities += len(batch)
except Exception as e:
print(f" Error batch {label}: {str(e)[:80]}")
print(f" {label}: {len(ents)} entidades")
# Subir relaciones MENCIONA en batch (chunk -> entidad)
print(f" Conectando chunks con entidades (MENCIONA)...")
menciona_batch = []
for ent in entities:
label = TYPE_TO_LABEL.get(ent["tipo"], "Entidad") + suffix
for chunk_id in ent.get("chunk_ids", [])[:10]:
if chunk_id:
menciona_batch.append({
"nombre": ent["nombre"],
"label": label,
"chunk_id": chunk_id,
})
# MENCIONA necesita match por label, agrupar por label
menciona_by_label = {}
for m in menciona_batch:
if m["label"] not in menciona_by_label:
menciona_by_label[m["label"]] = []
menciona_by_label[m["label"]].append(m)
total_menciona = 0
for label, items in menciona_by_label.items():
for i in range(0, len(items), BATCH):
batch = [{"nombre": m["nombre"], "chunk_id": m["chunk_id"]} for m in items[i:i + BATCH]]
try:
run_write(f"""
UNWIND $batch AS m
MATCH (e:{label} {{nombre: m.nombre}})
MATCH (c:Chunk {{id: m.chunk_id}})
MERGE (c)-[:MENCIONA]->(e)
""", {"batch": batch})
total_menciona += len(batch)
except Exception as e:
print(f" Error MENCIONA {label}: {str(e)[:80]}")
print(f" {total_menciona} relaciones MENCIONA")
# Subir relaciones entre entidades en batch, agrupadas por tipo de relacion
print(f" Subiendo {len(relations)} relaciones entre entidades...")
# Pre-build lookup de nombre -> label
entity_label = {}
for ent in entities:
entity_label[ent["nombre"]] = TYPE_TO_LABEL.get(ent["tipo"], "Entidad") + suffix
# Agrupar por (desde_label, relacion, hasta_label)
rel_groups = {}
for rel in relations:
desde_l = entity_label.get(rel["desde"])
hasta_l = entity_label.get(rel["hasta"])
if desde_l and hasta_l:
key = (desde_l, rel["relacion"], hasta_l)
if key not in rel_groups:
rel_groups[key] = []
rel_groups[key].append({"desde": rel["desde"], "hasta": rel["hasta"]})
for (desde_l, rel_type, hasta_l), items in rel_groups.items():
for i in range(0, len(items), BATCH):
batch = items[i:i + BATCH]
try:
run_write(f"""
UNWIND $batch AS r
MATCH (a:{desde_l} {{nombre: r.desde}})
MATCH (b:{hasta_l} {{nombre: r.hasta}})
MERGE (a)-[:{rel_type}]->(b)
""", {"batch": batch})
created_relations += len(batch)
except Exception as e:
print(f" Error {desde_l}-[{rel_type}]->{hasta_l}: {str(e)[:80]}")
print(f" Resultado: {created_entities} entidades, {total_menciona} MENCIONA, {created_relations} relaciones")
return {"entities": created_entities, "menciona": total_menciona, "relations": created_relations}
def promote_dev_entities():
"""Renombra labels Dev a produccion (ej: PatologiaDev -> Patologia)."""
for tipo, label in TYPE_TO_LABEL.items():
dev_label = label + "Dev"
count = run_query(f"MATCH (n:{dev_label}) RETURN count(n) AS n")
n = count[0]["n"] if count else 0
if n > 0:
print(f" Promoviendo {n} nodos {dev_label} -> {label}...")
run_write(f"MATCH (n:{dev_label}) SET n:{label} REMOVE n:{dev_label}")
print(" Promocion completa.")
# --- CLI ---
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print("\nextract_entities -- Extraccion masiva de entidades medicas con LLM")
print("\nUso:")
print(" python extract_entities.py <libro_id> - Extraer todo")
print(" python extract_entities.py <libro_id> --preview - Solo 10 chunks, sin subir")
print(" python extract_entities.py <libro_id> --limit 50 - Solo N chunks")
print(" python extract_entities.py <libro_id> --skip 30 - Saltear primeros N")
print(" python extract_entities.py <libro_id> --dev - Subir con labels Dev")
print(" python extract_entities.py <libro_id> --upload - Subir extracciones guardadas")
print(" python extract_entities.py promote - Mover Dev -> produccion")
print(" python extract_entities.py stats - Ver stats del grafo")
sys.exit(0)
cmd = sys.argv[1]
args = sys.argv[2:]
if cmd == "promote":
promote_dev_entities()
sys.exit(0)
if cmd == "stats":
for tipo, label in TYPE_TO_LABEL.items():
for suffix in ["", "Dev"]:
full_label = label + suffix
count = run_query(f"MATCH (n:{full_label}) RETURN count(n) AS n")
n = count[0]["n"] if count else 0
if n > 0:
print(f" {full_label}: {n}")
# Relaciones
rels = run_query("MATCH ()-[r:MENCIONA]->() RETURN count(r) AS n")
print(f" MENCIONA: {rels[0]['n'] if rels else 0}")
sys.exit(0)
libro_id = cmd
preview = "--preview" in args
dev_mode = "--dev" in args or preview
upload_only = "--upload" in args
limit = 0
if "--limit" in args:
idx = args.index("--limit")
limit = int(args[idx + 1]) if idx + 1 < len(args) else 10
skip = 0
if "--skip" in args:
idx = args.index("--skip")
skip = int(args[idx + 1]) if idx + 1 < len(args) else 0
if preview:
limit = limit or 10
if upload_only:
# Solo subir extracciones ya guardadas
output_path = os.path.join(OUTPUT_DIR, f"{libro_id}_entities.json")
if not os.path.exists(output_path):
print(f"No hay extracciones guardadas para {libro_id}")
sys.exit(1)
with open(output_path, "r", encoding="utf-8") as f:
data = json.load(f)
entities, relations = canonicalize_entities(data["extractions"])
print(f" Canonicalizadas: {len(entities)} entidades, {len(relations)} relaciones")
upload_entities(entities, relations, libro_id, dev_mode=dev_mode)
sys.exit(0)
# Extraer
print(f"\n{'='*60}")
print(f" EXTRACCION: {libro_id}")
print(f" Modo: {'PREVIEW' if preview else 'DEV' if dev_mode else 'PRODUCCION'}")
print(f" Chunks: {'todos' if not limit else limit}")
print(f"{'='*60}\n")
stats = extract_libro(libro_id, limit=limit, skip_first=skip)
# Canonicalizar
output_path = stats["output_path"]
with open(output_path, "r", encoding="utf-8") as f:
data = json.load(f)
entities, relations = canonicalize_entities(data["extractions"])
print(f"\n Canonicalizadas: {len(entities)} entidades unicas, {len(relations)} relaciones unicas")
# Top 20 entidades por frecuencia
print(f"\n Top 20 entidades:")
for ent in entities[:20]:
sins = f" ({', '.join(ent['sinonimos'][:3])})" if ent.get('sinonimos') else ""
print(f" [{ent['tipo']}] {ent['nombre']}{sins} (freq: {ent['freq']})")
if not preview:
print(f"\n Subiendo a Neo4j...")
upload_entities(entities, relations, libro_id, dev_mode=dev_mode)

227
load_dags.py Normal file
View File

@ -0,0 +1,227 @@
"""Carga DAGs desde YAML a Neo4j como relaciones PATHWAY y CLINICAL.
Los nodos referenciados deben existir en el grafo. Si no existen,
se crean como nodos genéricos con el tipo indicado.
Uso:
python load_dags.py # Cargar todos los YAML de dags/
python load_dags.py dags/otitis.yaml # Cargar uno específico
python load_dags.py --list # Listar DAGs cargados
python load_dags.py --delete nombre # Borrar un DAG
"""
import os
import sys
import yaml
from db import run_write, run_query
DAGS_DIR = os.path.join(os.path.dirname(__file__), "dags")
# Mapa tipo_nodo -> label Neo4j
TIPO_TO_LABEL = {
"patologia": "Patologia",
"sintoma": "Sintoma",
"signo": "Signo",
"procedimiento": "Procedimiento",
"farmaco": "Farmaco",
"agente": "Agente",
"hallazgo": "Hallazgo",
"metodo_dx": "MetodoDx",
"parametro": "Parametro",
"estructura_anatomica": "EstructuraAnatomica",
"grupo_farmacologico": "GrupoFarmacologico",
}
def load_yaml(path: str) -> dict:
with open(path, "r", encoding="utf-8") as f:
return yaml.safe_load(f)
def ensure_node_exists(nombre: str, tipo_nodo: str):
"""Si el nodo no existe en el grafo, lo crea."""
label = TIPO_TO_LABEL.get(tipo_nodo, "Hallazgo")
existing = run_query(f"""
MATCH (n:{label})
WHERE toLower(n.nombre) = toLower($nombre)
RETURN n.nombre AS nombre LIMIT 1
""", {"nombre": nombre})
if not existing:
# Buscar en cualquier label
any_match = run_query("""
MATCH (n)
WHERE toLower(n.nombre) = toLower($nombre)
AND NOT n:Chunk AND NOT n:ParentChunk
RETURN n.nombre AS nombre, labels(n)[0] AS label LIMIT 1
""", {"nombre": nombre})
if not any_match:
# Crear nodo nuevo
run_write(f"""
CREATE (n:{label} {{nombre: $nombre, tipo: $tipo, fuente: 'dag'}})
""", {"nombre": nombre, "tipo": tipo_nodo})
return "created"
return "found_other_label"
return "exists"
def load_dag(dag: dict) -> dict:
"""Carga un DAG a Neo4j."""
nombre_dag = dag["nombre"]
tipo_dag = dag["tipo"] # pathway o clinical
rel_type = "PATHWAY" if tipo_dag == "pathway" else "CLINICAL"
pasos = dag["pasos"]
print(f" Cargando DAG: {nombre_dag} ({tipo_dag}, {len(pasos)} pasos)")
# Borrar DAG existente con mismo nombre
run_write(f"""
MATCH ()-[r:{rel_type} {{nombre_dag: $nombre}}]->()
DELETE r
""", {"nombre": nombre_dag})
# Asegurar que todos los nodos existen
for paso in pasos:
status = ensure_node_exists(paso["nodo"], paso["tipo_nodo"])
if status == "created":
print(f" Nodo creado: {paso['nodo']} ({paso['tipo_nodo']})")
# Crear relaciones entre pasos consecutivos
created = 0
for i in range(len(pasos) - 1):
current = pasos[i]
next_paso = pasos[i + 1]
# Si tienen el mismo orden, son bifurcaciones del paso anterior
# Conectar desde el último paso con orden menor
if current["orden"] == next_paso["orden"]:
continue
current_label = TIPO_TO_LABEL.get(current["tipo_nodo"], "Hallazgo")
next_label = TIPO_TO_LABEL.get(next_paso["tipo_nodo"], "Hallazgo")
props = {
"nombre_dag": nombre_dag,
"orden": next_paso["orden"],
"tipo_paso": next_paso.get("tipo_paso", ""),
"condicion": next_paso.get("condicion", ""),
"nota": next_paso.get("nota", ""),
}
try:
run_write(f"""
MATCH (a:{current_label}) WHERE toLower(a.nombre) = toLower($from_name)
MATCH (b:{next_label}) WHERE toLower(b.nombre) = toLower($to_name)
CREATE (a)-[:{rel_type} {{
nombre_dag: $props.nombre_dag,
orden: $props.orden,
tipo_paso: $props.tipo_paso,
condicion: $props.condicion,
nota: $props.nota
}}]->(b)
""", {"from_name": current["nodo"], "to_name": next_paso["nodo"], "props": props})
created += 1
except Exception as e:
print(f" Error: {current['nodo']} -> {next_paso['nodo']}: {str(e)[:60]}")
# Para bifurcaciones (mismo orden), conectar desde el paso anterior
orders = sorted(set(p["orden"] for p in pasos))
for idx, order in enumerate(orders):
pasos_at_order = [p for p in pasos if p["orden"] == order]
if len(pasos_at_order) > 1:
# Encontrar el último paso del orden anterior
prev_order = orders[idx - 1] if idx > 0 else None
if prev_order is not None:
prev_pasos = [p for p in pasos if p["orden"] == prev_order]
source = prev_pasos[-1] # Último del orden anterior
source_label = TIPO_TO_LABEL.get(source["tipo_nodo"], "Hallazgo")
for branch in pasos_at_order:
branch_label = TIPO_TO_LABEL.get(branch["tipo_nodo"], "Hallazgo")
props = {
"nombre_dag": nombre_dag,
"orden": branch["orden"],
"tipo_paso": branch.get("tipo_paso", "decision"),
"condicion": branch.get("condicion", ""),
"nota": branch.get("nota", ""),
}
try:
run_write(f"""
MATCH (a:{source_label}) WHERE toLower(a.nombre) = toLower($from_name)
MATCH (b:{branch_label}) WHERE toLower(b.nombre) = toLower($to_name)
CREATE (a)-[:{rel_type} {{
nombre_dag: $props.nombre_dag,
orden: $props.orden,
tipo_paso: $props.tipo_paso,
condicion: $props.condicion,
nota: $props.nota
}}]->(b)
""", {"from_name": source["nodo"], "to_name": branch["nodo"], "props": props})
created += 1
except Exception as e:
print(f" Error bifurcacion: {source['nodo']} -> {branch['nodo']}: {str(e)[:60]}")
print(f" Resultado: {created} relaciones {rel_type}")
return {"dag": nombre_dag, "relations": created}
def list_dags():
"""Lista DAGs cargados en Neo4j."""
for rel_type in ["PATHWAY", "CLINICAL"]:
dags = run_query(f"""
MATCH ()-[r:{rel_type}]->()
RETURN DISTINCT r.nombre_dag AS nombre, count(r) AS relaciones
ORDER BY nombre
""")
if dags:
print(f"\n {rel_type}:")
for d in dags:
print(f" {d['nombre']}: {d['relaciones']} relaciones")
def delete_dag(nombre: str):
"""Borra un DAG por nombre."""
for rel_type in ["PATHWAY", "CLINICAL"]:
run_write(f"""
MATCH ()-[r:{rel_type} {{nombre_dag: $nombre}}]->()
DELETE r
""", {"nombre": nombre})
print(f" DAG '{nombre}' borrado")
# CLI
if __name__ == "__main__":
if len(sys.argv) < 2:
# Cargar todos los YAML
if not os.path.exists(DAGS_DIR):
print("No existe directorio dags/")
sys.exit(1)
yamls = [f for f in os.listdir(DAGS_DIR) if f.endswith((".yaml", ".yml"))]
if not yamls:
print("No hay archivos YAML en dags/")
sys.exit(1)
print(f"Cargando {len(yamls)} DAGs...")
for fname in sorted(yamls):
dag = load_yaml(os.path.join(DAGS_DIR, fname))
load_dag(dag)
print("\nDAGs cargados:")
list_dags()
elif sys.argv[1] == "--list":
list_dags()
elif sys.argv[1] == "--delete" and len(sys.argv) > 2:
delete_dag(sys.argv[2])
else:
path = sys.argv[1]
if os.path.exists(path):
dag = load_yaml(path)
load_dag(dag)
else:
print(f"Archivo no encontrado: {path}")

161
mcp_server.py Normal file
View File

@ -0,0 +1,161 @@
import os
"""MCP Server para MedGraph — conecta Claude directamente a la base de conocimiento medica."""
import json
import httpx
from mcp.server.fastmcp import FastMCP
API_URL = os.getenv("MEDGRAPH_API_URL", "http://localhost:8000")
API_KEY = os.getenv("API_KEY", "")
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
mcp = FastMCP(
"MedGraph",
instructions="""Sos un asistente de estudio medico. Tenes acceso a MedGraph, una base de conocimiento
con bibliografia indexada, un grafo de conceptos medicos interconectados, y material academico.
Reglas:
- SIEMPRE usar medgraph_query como primera opcion. Es el endpoint unificado que analiza la pregunta y activa automaticamente las capas necesarias (ontologia, grafo, bibliografia, actividades, flujos clinicos).
- Solo usar las tools especificas (medgraph_search, medgraph_activity, etc.) si necesitas algo muy puntual que medgraph_query no cubrio.
- Siempre citar fuentes (libro, paginas).
- No inventar. Si no encontras datos, decirlo.
- Responder en español.""",
)
async def _api_get(path: str) -> dict:
async with httpx.AsyncClient(verify=True, timeout=60) as client:
r = await client.get(f"{API_URL}{path}", headers=HEADERS)
r.raise_for_status()
return r.json()
async def _api_post(path: str, body: dict) -> dict:
async with httpx.AsyncClient(verify=True, timeout=60) as client:
r = await client.post(f"{API_URL}{path}", headers=HEADERS, json=body)
r.raise_for_status()
return r.json()
@mcp.tool()
async def medgraph_query(pregunta: str, top_k: int = 8) -> str:
"""Consulta inteligente unificada a MedGraph. Analiza la pregunta automaticamente
y activa las capas necesarias: ontologia (clasificacion ATC/SNOMED), grafo de
conocimiento (relaciones entre entidades), bibliografia (busqueda en libros),
actividades academicas (TPs, seminarios), y flujos clinicos (DAGs).
USAR SIEMPRE COMO PRIMERA OPCION para cualquier pregunta medica."""
data = await _api_post("/query", {"pregunta": pregunta, "top_k": top_k})
return json.dumps(data, ensure_ascii=False, indent=2)
@mcp.tool()
async def medgraph_comprehensive(tema: str) -> str:
"""Obtener TODO sobre un tema medico: grafo de conocimiento, actividades academicas,
course materials y bibliografia con citas exactas. Usar para preguntas amplias
como 'contame sobre otitis', 'todo sobre HTA', 'preparame para el TP de otoscopia'."""
data = await _api_get(f"/topic/{tema}/comprehensive")
return json.dumps(data, ensure_ascii=False, indent=2)
@mcp.tool()
async def medgraph_search(query: str, top_k: int = 5, libro_id: str = "") -> str:
"""Buscar en la bibliografia medica indexada. Busqueda hibrida (semantica + keywords).
Usar para preguntas puntuales como 'dosis de amoxicilina', 'valores normales de hemograma',
'definicion de fovea'. Libros disponibles: farreras-2020, harrison-manual, garcia-feijoo-oftalmo,
diamante-orl, balcells-laboratorio, goodman-gilman-farma, sanguinetti-semiologia,
and more. Configure your own books via the pipeline.
your indexed books appear here after running the pipeline.
"""
body = {"query": query, "top_k": top_k}
if libro_id:
body["libro_id"] = libro_id
data = await _api_post("/search/hybrid", body)
return json.dumps(data, ensure_ascii=False, indent=2)
@mcp.tool()
async def medgraph_pathways(tema: str) -> str:
"""Obtener la cadena causal/fisiopatologica de un tema. Secuencia determinista:
agente -> mecanismo -> efecto -> signo -> complicacion.
Usar cuando pregunten 'por que se produce X', 'fisiopatologia de X', 'mecanismo de X'."""
data = await _api_get(f"/topic/{tema}/pathways")
return json.dumps(data, ensure_ascii=False, indent=2)
@mcp.tool()
async def medgraph_clinical(tema: str) -> str:
"""Obtener el arbol de decision clinica ante un tema. Bifurcaciones con condiciones.
Usar cuando pregunten 'que hago ante un paciente con X', 'como manejo X',
'diagnostico diferencial de X'."""
data = await _api_get(f"/topic/{tema}/clinical")
return json.dumps(data, ensure_ascii=False, indent=2)
@mcp.tool()
async def medgraph_activity(nombre: str) -> str:
"""Buscar una actividad academica (TP, Seminario, Taller, Acreditacion) por nombre.
Usar cuando mencionen un TP, seminario o taller especifico."""
data = await _api_get(f"/activity/search/{nombre}")
return json.dumps(data, ensure_ascii=False, indent=2)
@mcp.tool()
async def medgraph_activity_material(activity_id: str) -> str:
"""Obtener el contenido completo de los documentos de una actividad (guias, procedimientos).
Usar despues de medgraph_activity para leer el material."""
data = await _api_get(f"/activity/{activity_id}/material")
return json.dumps(data, ensure_ascii=False, indent=2)
@mcp.tool()
async def medgraph_pathology(nombre: str) -> str:
"""Informacion estructurada de una patologia del grafo: definicion, diagnostico,
tratamiento, signos, agentes, fuentes bibliograficas."""
data = await _api_get(f"/pathology/{nombre}")
return json.dumps(data, ensure_ascii=False, indent=2)
@mcp.tool()
async def medgraph_procedure(nombre: str) -> str:
"""Pasos, insumos, hallazgos y parametros de un procedimiento medico."""
data = await _api_get(f"/procedure/{nombre}")
return json.dumps(data, ensure_ascii=False, indent=2)
@mcp.tool()
async def medgraph_ontology(tema: str) -> str:
"""Clasificacion ontologica de un tema medico: jerarquia ATC de farmacos,
jerarquia SNOMED de patologias/anatomia/procedimientos, y queries cruzadas
entre sistemas farmacologicos y sistemas corporales.
Usar cuando pregunten 'a que clase pertenece X', 'que tipo de farmaco es X',
'que sistema afecta X', 'farmacos de clase X para patologias de sistema Y'."""
data = await _api_get(f"/topic/{tema}/ontology")
return json.dumps(data, ensure_ascii=False, indent=2)
SCHEDULE_API_URL = os.getenv("SCHEDULE_API_URL", "http://localhost:8001")
SCHEDULE_API_KEY = os.getenv("SCHEDULE_API_KEY", "")
SCHEDULE_HEADERS = {"X-Medgraph-Key": SCHEDULE_API_KEY}
@mcp.tool()
async def medgraph_cronograma(semana_numero: int = 0) -> str:
"""Obtener el cronograma semanal de la carrera de medicina. Incluye todas las
actividades con horarios, tipo, area, si es obligatoria,
docente y lugar. Si semana_numero es 0, devuelve la semana actual."""
async with httpx.AsyncClient(verify=True, timeout=60) as client:
params = {"group": "A", "subgroup": "A1"}
if semana_numero > 0:
params["semana_numero"] = semana_numero
r = await client.get(
f"{SCHEDULE_API_URL}/api/v1/cronograma" # Configure your schedule API,
headers=SCHEDULE_HEADERS,
params=params,
)
r.raise_for_status()
return json.dumps(r.json(), ensure_ascii=False, indent=2)
if __name__ == "__main__":
mcp.run(transport="stdio")

431
migrate_chunks.py Normal file
View File

@ -0,0 +1,431 @@
"""Migración de chunks v1 -> v2 en Neo4j.
Módulo importable. Funciones principales:
- upload_chunks_for_libro(libro_id, children, parents) — sube chunks de un libro
- delete_libro_chunks(libro_id) — borra chunks de un libro
- normalize_chunks(children) — agrega text_busqueda, keywords, etc.
- create_relationships(libro_id) — crea CHILD_OF y SIGUE_A
- full_migration() — migración global v1 -> v2 con swap de labels
CLI: python migrate_chunks.py [upload|swap|verify|clean|full]
"""
import json
import os
import unicodedata
import time
from db import run_write, run_query
PARSED_DIR = os.path.join(os.path.dirname(__file__), "parsed")
CATALOG_PATH = os.path.join(os.path.dirname(__file__), "catalog.json")
BATCH_SIZE = 100
def normalize_for_search(text: str) -> str:
"""Quita acentos y pasa a minúsculas para full-text search."""
if not text:
return ""
nfkd = unicodedata.normalize('NFKD', text)
return ''.join(c for c in nfkd if not unicodedata.combining(c)).lower()
def normalize_chunks(children: list) -> None:
"""Agrega propiedades normalizadas a los chunks (in-place).
Agrega: text_busqueda, titulo_seccion_busqueda, titulo_capitulo_busqueda, keywords
"""
for chunk in children:
chunk["text_busqueda"] = normalize_for_search(chunk["text"])
chunk["titulo_seccion_busqueda"] = normalize_for_search(chunk.get("titulo_seccion", ""))
chunk["titulo_capitulo_busqueda"] = normalize_for_search(chunk.get("titulo_capitulo", ""))
# Keywords: combinar título + primeras 50 palabras
kw_parts = []
if chunk.get("titulo_capitulo"):
kw_parts.append(chunk["titulo_capitulo"])
if chunk.get("titulo_seccion"):
kw_parts.append(chunk["titulo_seccion"])
first_words = " ".join(chunk["text"].split()[:50])
kw_parts.append(first_words)
chunk["keywords"] = normalize_for_search(" ".join(kw_parts))
def upload_chunks_for_libro(libro_id: str, children: list, parents: list,
label: str = "Chunk",
on_progress: callable = None) -> dict:
"""Sube children y parent chunks de un libro a Neo4j.
Args:
libro_id: ID del libro
children: Lista de child chunks
parents: Lista de parent chunks
label: Label para children (default "Chunk", usar "ChunkV2" para migración)
on_progress: Callback(step, pct, msg)
Returns:
dict con estadísticas
"""
def report(pct, msg):
if on_progress:
on_progress("upload", pct, msg)
print(f" [{pct}%] {msg}")
total_children = len(children)
total_parents = len(parents)
# Subir children en batches
report(0, f"Subiendo {total_children} children como :{label}...")
for i in range(0, total_children, BATCH_SIZE):
batch = children[i:i + BATCH_SIZE]
_upload_children_batch(batch, label)
pct = min(60, int((i + len(batch)) / total_children * 60))
if (i + BATCH_SIZE) % 500 < BATCH_SIZE:
report(pct, f" {i + len(batch)}/{total_children} children")
# Subir parents en batches
report(60, f"Subiendo {total_parents} parents como :ParentChunk...")
for i in range(0, total_parents, BATCH_SIZE):
batch = parents[i:i + BATCH_SIZE]
_upload_parents_batch(batch)
pct = 60 + min(30, int((i + len(batch)) / total_parents * 30))
if (i + BATCH_SIZE) % 500 < BATCH_SIZE:
report(pct, f" {i + len(batch)}/{total_parents} parents")
report(90, "Upload completo")
return {"children": total_children, "parents": total_parents}
def _upload_children_batch(batch: list, label: str = "Chunk"):
"""Sube un batch de child chunks."""
# Usar CREATE en vez de MERGE para velocidad (asumimos IDs únicos)
cypher = f"""
UNWIND $batch AS chunk
CREATE (c:{label} {{
id: chunk.id,
libro_id: chunk.libro_id,
text: chunk.text,
page_start: chunk.page_start,
page_end: chunk.page_end,
word_count: chunk.word_count,
titulo_capitulo: chunk.titulo_capitulo,
titulo_seccion: chunk.titulo_seccion,
tipo_contenido: chunk.tipo_contenido,
parent_id: chunk.parent_id,
chunk_index: chunk.chunk_index,
version: chunk.version,
text_busqueda: chunk.text_busqueda,
titulo_seccion_busqueda: chunk.titulo_seccion_busqueda,
titulo_capitulo_busqueda: chunk.titulo_capitulo_busqueda,
keywords: chunk.keywords
}})
"""
# Preparar batch con defaults para propiedades que podrían faltar
safe_batch = []
for c in batch:
safe_batch.append({
"id": c["id"],
"libro_id": c["libro_id"],
"text": c["text"],
"page_start": c["page_start"],
"page_end": c["page_end"],
"word_count": c["word_count"],
"titulo_capitulo": c.get("titulo_capitulo", ""),
"titulo_seccion": c.get("titulo_seccion", ""),
"tipo_contenido": c.get("tipo_contenido", "body"),
"parent_id": c.get("parent_id", ""),
"chunk_index": c.get("chunk_index", 0),
"version": c.get("version", 2),
"text_busqueda": c.get("text_busqueda", ""),
"titulo_seccion_busqueda": c.get("titulo_seccion_busqueda", ""),
"titulo_capitulo_busqueda": c.get("titulo_capitulo_busqueda", ""),
"keywords": c.get("keywords", ""),
})
run_write(cypher, {"batch": safe_batch})
def _upload_parents_batch(batch: list):
"""Sube un batch de parent chunks."""
run_write("""
UNWIND $batch AS parent
CREATE (p:ParentChunk {
id: parent.id,
libro_id: parent.libro_id,
text: parent.text,
page_start: parent.page_start,
page_end: parent.page_end,
word_count: parent.word_count,
titulo_capitulo: parent.titulo_capitulo,
titulo_seccion: parent.titulo_seccion
})
""", {"batch": [{
"id": p["id"],
"libro_id": p["libro_id"],
"text": p["text"],
"page_start": p["page_start"],
"page_end": p["page_end"],
"word_count": p["word_count"],
"titulo_capitulo": p.get("titulo_capitulo", ""),
"titulo_seccion": p.get("titulo_seccion", ""),
} for p in batch]})
def create_relationships(libro_id: str = None, label: str = "Chunk"):
"""Crea relaciones CHILD_OF y SIGUE_A para chunks de un libro (o todos).
Args:
libro_id: Si se da, solo para ese libro. Si None, para todos.
label: Label de los children (default "Chunk")
"""
libro_filter = "AND child.libro_id = $libro_id" if libro_id else ""
params = {"libro_id": libro_id} if libro_id else {}
# CHILD_OF
print(f" Creando relaciones CHILD_OF{f' para {libro_id}' if libro_id else ''}...")
run_write(f"""
MATCH (child:{label})
WHERE child.parent_id IS NOT NULL AND child.parent_id <> '' {libro_filter}
WITH child
MATCH (parent:ParentChunk {{id: child.parent_id}})
CREATE (child)-[:CHILD_OF]->(parent)
""", params)
# SIGUE_A (chunks consecutivos del mismo libro)
print(f" Creando relaciones SIGUE_A{f' para {libro_id}' if libro_id else ''}...")
if libro_id:
run_write(f"""
MATCH (c1:{label} {{libro_id: $libro_id}})
WITH c1 ORDER BY c1.chunk_index
WITH collect(c1) AS chunks
UNWIND range(0, size(chunks)-2) AS i
WITH chunks[i] AS c1, chunks[i+1] AS c2
CREATE (c1)-[:SIGUE_A]->(c2)
""", {"libro_id": libro_id})
else:
# Para todos: agrupar por libro
libros = run_query(f"MATCH (c:{label}) RETURN DISTINCT c.libro_id AS lid")
for row in libros:
lid = row["lid"]
print(f" SIGUE_A para {lid}...")
run_write(f"""
MATCH (c1:{label} {{libro_id: $libro_id}})
WITH c1 ORDER BY c1.chunk_index
WITH collect(c1) AS chunks
UNWIND range(0, size(chunks)-2) AS i
WITH chunks[i] AS c1, chunks[i+1] AS c2
CREATE (c1)-[:SIGUE_A]->(c2)
""", {"libro_id": lid})
def delete_libro_chunks(libro_id: str):
"""Borra todos los chunks y parents de un libro específico."""
print(f" Borrando chunks de {libro_id}...")
# Borrar children
run_write("""
MATCH (c:Chunk {libro_id: $libro_id})
DETACH DELETE c
""", {"libro_id": libro_id})
# Borrar parents
run_write("""
MATCH (p:ParentChunk {libro_id: $libro_id})
DETACH DELETE p
""", {"libro_id": libro_id})
print(f" Borrado completo para {libro_id}")
def verify_counts():
"""Verifica conteo de chunks en Neo4j."""
results = {}
# Chunks v1
v1 = run_query("MATCH (c:Chunk) RETURN count(c) AS n")
results["chunks"] = v1[0]["n"] if v1 else 0
# Chunks v2 (si existen durante migración)
v2 = run_query("MATCH (c:ChunkV2) RETURN count(c) AS n")
results["chunks_v2"] = v2[0]["n"] if v2 else 0
# Parents
parents = run_query("MATCH (p:ParentChunk) RETURN count(p) AS n")
results["parents"] = parents[0]["n"] if parents else 0
# ChunkV1 (backup durante swap)
v1_old = run_query("MATCH (c:ChunkV1) RETURN count(c) AS n")
results["chunks_v1_backup"] = v1_old[0]["n"] if v1_old else 0
# Relaciones
child_of = run_query("MATCH ()-[r:CHILD_OF]->() RETURN count(r) AS n")
results["child_of_rels"] = child_of[0]["n"] if child_of else 0
sigue_a = run_query("MATCH ()-[r:SIGUE_A]->() RETURN count(r) AS n")
results["sigue_a_rels"] = sigue_a[0]["n"] if sigue_a else 0
# Por libro
by_libro = run_query("MATCH (c:Chunk) RETURN c.libro_id AS libro, count(c) AS n ORDER BY n DESC")
results["by_libro"] = {r["libro"]: r["n"] for r in by_libro}
return results
def full_migration():
"""Migración completa v1 -> v2 con swap de labels.
Pasos:
1. Cargar y normalizar todos los chunks v2 de parsed/
2. Subir como :ChunkV2 + :ParentChunk
3. Verificar conteo
4. Swap: Chunk -> ChunkV1, ChunkV2 -> Chunk
5. Crear relaciones
6. Limpiar ChunkV1
"""
print("=" * 60)
print(" MIGRACIÓN COMPLETA v1 -> v2")
print("=" * 60)
catalog = json.load(open(CATALOG_PATH, "r", encoding="utf-8"))
# Paso 1: Cargar todos los chunks v2
all_children = []
all_parents = []
for libro in catalog["libros"]:
children_path = os.path.join(PARSED_DIR, f"{libro['id']}_v2_chunks.json")
parents_path = os.path.join(PARSED_DIR, f"{libro['id']}_v2_parents.json")
if not os.path.exists(children_path):
print(f" SKIP: {libro['titulo']} (no tiene v2 chunks)")
continue
with open(children_path, "r", encoding="utf-8") as f:
children = json.load(f)
with open(parents_path, "r", encoding="utf-8") as f:
parents = json.load(f)
# Normalizar para full-text
normalize_chunks(children)
all_children.extend(children)
all_parents.extend(parents)
print(f" Cargado: {libro['titulo']} — {len(children)} children, {len(parents)} parents")
print(f"\n Total a subir: {len(all_children)} children, {len(all_parents)} parents")
# Paso 2: Subir como ChunkV2
print("\n Paso 2: Subiendo chunks v2...")
upload_chunks_for_libro("_all_", all_children, all_parents, label="ChunkV2")
# Paso 3: Verificar conteo
print("\n Paso 3: Verificando...")
counts = verify_counts()
print(f" Chunks v1 actuales: {counts['chunks']}")
print(f" Chunks v2 nuevos: {counts['chunks_v2']}")
print(f" Parents nuevos: {counts['parents']}")
if counts['chunks_v2'] == 0:
print(" ERROR: No se subieron chunks v2. Abortando.")
return
# Paso 4: Swap de labels
print("\n Paso 4: Swap de labels...")
print(" Chunk -> ChunkV1...")
run_write("MATCH (c:Chunk) SET c:ChunkV1 REMOVE c:Chunk")
time.sleep(1)
print(" ChunkV2 -> Chunk...")
run_write("MATCH (c:ChunkV2) SET c:Chunk REMOVE c:ChunkV2")
time.sleep(1)
# Paso 5: Crear relaciones
print("\n Paso 5: Creando relaciones...")
create_relationships(label="Chunk")
# Paso 6: Verificar
print("\n Paso 6: Verificación final...")
counts = verify_counts()
print(f" Chunks (nuevos): {counts['chunks']}")
print(f" Parents: {counts['parents']}")
print(f" CHILD_OF: {counts['child_of_rels']}")
print(f" SIGUE_A: {counts['sigue_a_rels']}")
print(f" ChunkV1 (backup): {counts['chunks_v1_backup']}")
print(f" Por libro: {counts['by_libro']}")
print("\n Migración completada. Los chunks v1 están en :ChunkV1 como backup.")
print(" Para limpiar: python migrate_chunks.py clean")
def clean_v1():
"""Borra los chunks v1 (backup post-swap)."""
counts = verify_counts()
v1_count = counts['chunks_v1_backup']
if v1_count == 0:
print(" No hay chunks v1 para limpiar.")
return
print(f" Borrando {v1_count} chunks v1 (backup)...")
run_write("MATCH (c:ChunkV1) DETACH DELETE c")
print(" Limpieza completada.")
# --- CLI ---
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print("\nmigrate_chunks — Migración de chunks v1 -> v2 en Neo4j")
print("\nUso:")
print(" python migrate_chunks.py verify - Ver estado actual")
print(" python migrate_chunks.py full - Migración completa v1 -> v2")
print(" python migrate_chunks.py clean - Borrar backup v1")
print(" python migrate_chunks.py libro <id> - Subir un libro específico (reemplaza)")
sys.exit(0)
cmd = sys.argv[1]
if cmd == "verify":
counts = verify_counts()
print(f"\n{'='*60}")
print(f" ESTADO DE NEO4J")
print(f"{'='*60}")
for k, v in counts.items():
if k != "by_libro":
print(f" {k}: {v}")
if counts.get("by_libro"):
print(f"\n Por libro:")
for libro, n in counts["by_libro"].items():
print(f" {libro}: {n}")
elif cmd == "full":
full_migration()
elif cmd == "clean":
clean_v1()
elif cmd == "libro" and len(sys.argv) > 2:
libro_id = sys.argv[2]
children_path = os.path.join(PARSED_DIR, f"{libro_id}_v2_chunks.json")
parents_path = os.path.join(PARSED_DIR, f"{libro_id}_v2_parents.json")
if not os.path.exists(children_path):
print(f"No se encontraron chunks v2 para '{libro_id}'")
print(f"Primero correr: python parser_v2.py {libro_id}")
sys.exit(1)
with open(children_path, "r", encoding="utf-8") as f:
children = json.load(f)
with open(parents_path, "r", encoding="utf-8") as f:
parents = json.load(f)
normalize_chunks(children)
delete_libro_chunks(libro_id)
upload_chunks_for_libro(libro_id, children, parents)
create_relationships(libro_id)
counts = verify_counts()
print(f"\n Verificación: {counts['by_libro'].get(libro_id, 0)} chunks para {libro_id}")
else:
print(f"Comando desconocido: {cmd}")

427
ontology.py Normal file
View File

@ -0,0 +1,427 @@
"""Ontología médica para MedGraph — ATC (fármacos) + SNOMED simplificado (patologías/anatomía).
Mapea entidades existentes del grafo a jerarquías taxonómicas usando Gemini LLM.
"""
import json
import os
import sys
import time
import re
from db import run_write, run_query
# Config
DATA_DIR = os.path.join(os.path.dirname(__file__), "data")
GEMINI_API_KEY = os.getenv("GCP_API_KEY", "")
def init_gemini():
"""Init Gemini via Google GenAI (same pattern as extract_entities.py)."""
from extract_entities import init_model
return init_model()
# ============================================================
# SEED: Cargar jerarquías estáticas a Neo4j
# ============================================================
def seed_atc_hierarchy(json_path=None):
"""Carga el árbol ATC como nodos CategoriaATC + relaciones ES_UN."""
if json_path is None:
json_path = os.path.join(DATA_DIR, "atc_hierarchy.json")
with open(json_path, "r", encoding="utf-8") as f:
tree = json.load(f)
nodes = []
relations = []
def walk(subtree, parent_code=None, nivel=1):
for code, data in subtree.items():
nombre = data["nombre"]
nodes.append({"codigo": code, "nombre": nombre, "nivel": nivel})
if parent_code:
relations.append({"hijo": code, "padre": parent_code})
if "children" in data and data["children"]:
walk(data["children"], code, nivel + 1)
walk(tree)
print(f" ATC: {len(nodes)} nodos, {len(relations)} relaciones ES_UN")
# Upload nodes in batch
for i in range(0, len(nodes), 200):
batch = nodes[i:i+200]
run_write("""
UNWIND $batch AS n
MERGE (c:CategoriaATC {codigo: n.codigo})
SET c.nombre = n.nombre, c.nivel = n.nivel
""", {"batch": batch})
# Upload ES_UN relations
for i in range(0, len(relations), 200):
batch = relations[i:i+200]
run_write("""
UNWIND $batch AS r
MATCH (hijo:CategoriaATC {codigo: r.hijo})
MATCH (padre:CategoriaATC {codigo: r.padre})
MERGE (hijo)-[:ES_UN]->(padre)
""", {"batch": batch})
print(f" ATC seeded: {len(nodes)} categorias")
return len(nodes)
def seed_snomed_hierarchy(json_path=None):
"""Carga el árbol SNOMED simplificado como nodos CategoriaSNOMED + relaciones ES_UN."""
if json_path is None:
json_path = os.path.join(DATA_DIR, "snomed_systems.json")
with open(json_path, "r", encoding="utf-8") as f:
tree = json.load(f)
nodes = []
relations = []
for sistema_code, sistema_data in tree.items():
# Nivel 1: Sistema
nodes.append({
"codigo": sistema_code,
"nombre": sistema_data["nombre"],
"nivel": 1,
"sistema": sistema_code
})
for subcat_code, subcat_data in sistema_data.get("subcategorias", {}).items():
# Nivel 2: Subcategoría
full_code = f"{sistema_code}_{subcat_code}"
nodes.append({
"codigo": full_code,
"nombre": subcat_data["nombre"],
"nivel": 2,
"sistema": sistema_code
})
relations.append({"hijo": full_code, "padre": sistema_code})
for grupo in subcat_data.get("grupos", []):
# Nivel 3: Grupo específico
grupo_code = f"{full_code}_{re.sub(r'[^a-z0-9]', '_', grupo.lower())[:30]}"
nodes.append({
"codigo": grupo_code,
"nombre": grupo.lower(),
"nivel": 3,
"sistema": sistema_code
})
relations.append({"hijo": grupo_code, "padre": full_code})
print(f" SNOMED: {len(nodes)} nodos, {len(relations)} relaciones ES_UN")
# Upload nodes
for i in range(0, len(nodes), 200):
batch = nodes[i:i+200]
run_write("""
UNWIND $batch AS n
MERGE (c:CategoriaSNOMED {codigo: n.codigo})
SET c.nombre = n.nombre, c.nivel = n.nivel, c.sistema = n.sistema
""", {"batch": batch})
# Upload ES_UN
for i in range(0, len(relations), 200):
batch = relations[i:i+200]
run_write("""
UNWIND $batch AS r
MATCH (hijo:CategoriaSNOMED {codigo: r.hijo})
MATCH (padre:CategoriaSNOMED {codigo: r.padre})
MERGE (hijo)-[:ES_UN]->(padre)
""", {"batch": batch})
print(f" SNOMED seeded: {len(nodes)} categorias")
return len(nodes)
# ============================================================
# MAPPING: Entidades existentes → Ontología con LLM
# ============================================================
def _llm_batch(model, prompt, max_retries=3):
"""Llama a Gemini (Google GenAI) y parsea JSON. Retry on failure."""
for attempt in range(max_retries):
try:
response = model.generate_content(
prompt,
generation_config={"temperature": 0.1, "max_output_tokens": 4096}
)
text = response.text.strip()
# Limpiar markdown
if text.startswith("```"):
text = re.sub(r'^```\w*\n?', '', text)
text = re.sub(r'\n?```$', '', text)
return json.loads(text)
except Exception as e:
if attempt < max_retries - 1:
time.sleep(2)
else:
print(f" LLM error after {max_retries} retries: {str(e)[:80]}")
return []
def map_farmacos_to_atc(batch_size=50, limit=0):
"""Mapea nodos Farmaco a códigos ATC usando Gemini."""
model = init_gemini()
# Get farmacos sin mapear
query = """
MATCH (f:Farmaco)
WHERE NOT (f)-[:ES_UN]->(:CategoriaATC)
RETURN f.nombre AS nombre
ORDER BY f.nombre
"""
if limit > 0:
query += f" LIMIT {limit}"
farmacos = run_query(query)
print(f" Farmacos sin mapear: {len(farmacos)}")
# Get valid ATC codes
atc_codes = run_query("MATCH (c:CategoriaATC) RETURN c.codigo AS codigo")
valid_codes = {r["codigo"] for r in atc_codes}
total_mapped = 0
total_skipped = 0
for i in range(0, len(farmacos), batch_size):
batch = farmacos[i:i+batch_size]
nombres = [f["nombre"] for f in batch]
prompt = f"""Dado estos nombres de farmacos, asigna el codigo ATC mas especifico posible.
Si no existe codigo exacto de nivel 5, asigna el nivel mas especifico que conozcas (nivel 3 o 4).
Si no es un farmaco real o no tiene codigo ATC, responde "SKIP".
Responde SOLO JSON array, sin texto adicional.
Formato: [{{"nombre": "...", "atc": "C03CA01"}}]
Farmacos: {json.dumps(nombres, ensure_ascii=False)}"""
results = _llm_batch(model, prompt)
if not results:
continue
mappings = []
for r in results:
nombre = r.get("nombre", "").lower().strip()
atc = r.get("atc", "SKIP").upper().strip()
if atc == "SKIP" or not nombre:
total_skipped += 1
continue
# Buscar el código ATC más cercano que exista en nuestra jerarquía
matched_code = None
for length in [7, 5, 4, 3, 1]: # De más específico a más general
candidate = atc[:length]
if candidate in valid_codes:
matched_code = candidate
break
if matched_code:
mappings.append({"nombre": nombre, "atc_code": matched_code})
else:
total_skipped += 1
# Upload batch
if mappings:
run_write("""
UNWIND $batch AS m
MATCH (f:Farmaco) WHERE toLower(f.nombre) = m.nombre
MATCH (c:CategoriaATC {codigo: m.atc_code})
MERGE (f)-[:ES_UN]->(c)
""", {"batch": mappings})
total_mapped += len(mappings)
pct = int((i + batch_size) / len(farmacos) * 100)
print(f" [{min(pct,100)}%] {total_mapped} mapeados, {total_skipped} skipped")
time.sleep(1)
print(f" ATC mapping done: {total_mapped} mapeados, {total_skipped} skipped")
return {"mapped": total_mapped, "skipped": total_skipped}
def map_entities_to_snomed(label, batch_size=30, limit=0):
"""Mapea entidades de un label a categorías SNOMED."""
model = init_gemini()
# Get sistemas válidos
sistemas = run_query("MATCH (c:CategoriaSNOMED {nivel: 1}) RETURN c.codigo AS codigo, c.nombre AS nombre")
sistema_list = [s["codigo"] for s in sistemas]
sistema_nombres = {s["codigo"]: s["nombre"] for s in sistemas}
# Get entidades sin mapear
query = f"""
MATCH (e:{label})
WHERE NOT (e)-[:ES_UN]->(:CategoriaSNOMED)
RETURN e.nombre AS nombre
ORDER BY e.nombre
"""
if limit > 0:
query += f" LIMIT {limit}"
entities = run_query(query)
print(f" {label} sin mapear: {len(entities)}")
# Get subcategorías válidas
subcats = run_query("MATCH (c:CategoriaSNOMED {nivel: 2}) RETURN c.codigo AS codigo, c.nombre AS nombre, c.sistema AS sistema")
subcat_map = {s["codigo"]: s for s in subcats}
total_mapped = 0
total_skipped = 0
tipo_texto = {
"Patologia": "patologias medicas",
"EstructuraAnatomica": "estructuras anatomicas del cuerpo humano",
"Procedimiento": "procedimientos medicos o diagnosticos"
}
for i in range(0, len(entities), batch_size):
batch = entities[i:i+batch_size]
nombres = [e["nombre"] for e in batch]
prompt = f"""Clasifica estos {tipo_texto.get(label, 'conceptos medicos')} en un sistema del cuerpo humano.
Sistemas validos: {json.dumps(sistema_list, ensure_ascii=False)}
Responde SOLO JSON array, sin texto adicional.
Formato: [{{"nombre": "...", "sistema": "cardiovascular", "subcategoria": "nombre descriptivo"}}]
Si no es un concepto medico real o no se puede clasificar, responde "SKIP" como sistema.
Conceptos: {json.dumps(nombres, ensure_ascii=False)}"""
results = _llm_batch(model, prompt)
if not results:
continue
mappings = []
for r in results:
nombre = r.get("nombre", "").lower().strip()
sistema = r.get("sistema", "SKIP").lower().strip()
if sistema == "skip" or not nombre or sistema not in sistema_list:
total_skipped += 1
continue
# Buscar la subcategoría más cercana
subcat_code = None
subcat_nombre = r.get("subcategoria", "").lower().strip()
# Intentar matchear por nombre de subcategoría
for code, data in subcat_map.items():
if data["sistema"] == sistema and (
subcat_nombre in data["nombre"].lower() or
data["nombre"].lower() in subcat_nombre
):
subcat_code = code
break
# Si no matchea subcategoría, vincular al sistema directamente
target_code = subcat_code if subcat_code else sistema
mappings.append({"nombre": nombre, "target": target_code})
if mappings:
run_write(f"""
UNWIND $batch AS m
MATCH (e:{label}) WHERE toLower(e.nombre) = m.nombre
MATCH (c:CategoriaSNOMED {{codigo: m.target}})
MERGE (e)-[:ES_UN]->(c)
""", {"batch": mappings})
total_mapped += len(mappings)
pct = int((i + batch_size) / len(entities) * 100)
print(f" [{min(pct,100)}%] {total_mapped} mapeados, {total_skipped} skipped")
time.sleep(1)
print(f" {label} SNOMED mapping done: {total_mapped} mapeados, {total_skipped} skipped")
return {"mapped": total_mapped, "skipped": total_skipped}
# ============================================================
# VALIDATE: Reportar cobertura
# ============================================================
def validate_mappings():
"""Reporta porcentaje de entidades mapeadas a ontología."""
labels = ["Farmaco", "Patologia", "EstructuraAnatomica", "Procedimiento"]
print("\n" + "=" * 60)
print(" COBERTURA ONTOLÓGICA")
print("=" * 60)
for label in labels:
total = run_query(f"MATCH (e:{label}) RETURN count(e) AS n")[0]["n"]
if label == "Farmaco":
mapped = run_query(f"MATCH (e:{label})-[:ES_UN]->(:CategoriaATC) RETURN count(DISTINCT e) AS n")[0]["n"]
else:
mapped = run_query(f"MATCH (e:{label})-[:ES_UN]->(:CategoriaSNOMED) RETURN count(DISTINCT e) AS n")[0]["n"]
pct = (mapped * 100 // total) if total > 0 else 0
bar = "█" * (pct // 5) + "░" * (20 - pct // 5)
print(f" {label:25s} {bar} {pct}% ({mapped}/{total})")
# Stats ontología
atc_nodes = run_query("MATCH (c:CategoriaATC) RETURN count(c) AS n")[0]["n"]
snomed_nodes = run_query("MATCH (c:CategoriaSNOMED) RETURN count(c) AS n")[0]["n"]
es_un = run_query("MATCH ()-[r:ES_UN]->() RETURN count(r) AS n")[0]["n"]
print(f"\n CategoriaATC: {atc_nodes} nodos")
print(f" CategoriaSNOMED: {snomed_nodes} nodos")
print(f" Relaciones ES_UN: {es_un}")
# ============================================================
# CLI
# ============================================================
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print("Uso: python ontology.py <comando> [opciones]")
print("Comandos:")
print(" seed-atc Cargar jerarquía ATC")
print(" seed-snomed Cargar jerarquía SNOMED")
print(" seed-all Cargar ambas jerarquías")
print(" map-farmacos Mapear Farmaco → ATC (--limit N)")
print(" map-patologias Mapear Patologia → SNOMED (--limit N)")
print(" map-anatomia Mapear EstructuraAnatomica → SNOMED (--limit N)")
print(" map-procedimientos Mapear Procedimiento → SNOMED (--limit N)")
print(" map-all Mapear todo (--limit N)")
print(" validate Reportar cobertura")
sys.exit(1)
cmd = sys.argv[1]
limit = 0
for i, arg in enumerate(sys.argv):
if arg == "--limit" and i + 1 < len(sys.argv):
limit = int(sys.argv[i + 1])
if cmd == "seed-atc":
seed_atc_hierarchy()
elif cmd == "seed-snomed":
seed_snomed_hierarchy()
elif cmd == "seed-all":
seed_atc_hierarchy()
seed_snomed_hierarchy()
elif cmd == "map-farmacos":
map_farmacos_to_atc(limit=limit)
elif cmd == "map-patologias":
map_entities_to_snomed("Patologia", limit=limit)
elif cmd == "map-anatomia":
map_entities_to_snomed("EstructuraAnatomica", limit=limit)
elif cmd == "map-procedimientos":
map_entities_to_snomed("Procedimiento", limit=limit)
elif cmd == "map-all":
print("=== Mapeando Farmacos → ATC ===")
map_farmacos_to_atc(limit=limit)
print("\n=== Mapeando Patologias → SNOMED ===")
map_entities_to_snomed("Patologia", limit=limit)
print("\n=== Mapeando EstructuraAnatomica → SNOMED ===")
map_entities_to_snomed("EstructuraAnatomica", limit=limit)
print("\n=== Mapeando Procedimientos → SNOMED ===")
map_entities_to_snomed("Procedimiento", limit=limit)
elif cmd == "validate":
validate_mappings()
else:
print(f"Comando desconocido: {cmd}")
sys.exit(1)

269
parser.py Normal file
View File

@ -0,0 +1,269 @@
"""Parser de PDFs a texto limpio + chunks semánticos."""
import fitz # PyMuPDF
import json
import os
import re
from datetime import datetime
LIBROS_DIR = os.path.join(os.path.dirname(__file__), "..", "LIBROS")
OUTPUT_DIR = os.path.join(os.path.dirname(__file__), "parsed")
CATALOG_PATH = os.path.join(os.path.dirname(__file__), "catalog.json")
def load_catalog():
with open(CATALOG_PATH, "r", encoding="utf-8") as f:
return json.load(f)
def save_catalog(catalog):
with open(CATALOG_PATH, "w", encoding="utf-8") as f:
json.dump(catalog, f, ensure_ascii=False, indent=2)
def parse_pdf(pdf_path: str, libro_id: str, chunk_size: int = 800) -> dict:
"""Parsea un PDF a texto limpio con chunks semánticos.
Args:
pdf_path: Ruta al PDF
libro_id: ID del libro en el catálogo
chunk_size: Tamaño objetivo de cada chunk en tokens (~palabras)
Returns:
dict con páginas y chunks
"""
doc = fitz.open(pdf_path)
total_pages = doc.page_count
print(f" Parseando {os.path.basename(pdf_path)} ({total_pages} pags)...")
# Extraer texto por página
pages = []
for i in range(total_pages):
page = doc[i]
text = page.get_text()
text = clean_text(text)
if len(text.strip()) > 20: # Ignorar páginas vacías/solo imagen
pages.append({
"page": i + 1,
"text": text
})
if (i + 1) % 500 == 0:
print(f" ... {i + 1}/{total_pages} paginas")
doc.close()
# Generar chunks semánticos
chunks = generate_chunks(pages, libro_id, chunk_size)
result = {
"libro_id": libro_id,
"archivo": os.path.basename(pdf_path),
"total_paginas": total_pages,
"paginas_con_texto": len(pages),
"chunks": len(chunks),
"fecha_parseo": datetime.now().isoformat(),
"pages_data": pages,
"chunks_data": chunks
}
print(f" Resultado: {len(pages)} pags con texto, {len(chunks)} chunks")
return result
def clean_text(text: str) -> str:
"""Limpia texto extraído de PDF."""
# Remover saltos de línea excesivos
text = re.sub(r'\n{3,}', '\n\n', text)
# Remover líneas que solo son números de página
text = re.sub(r'^\s*\d{1,4}\s*$', '', text, flags=re.MULTILINE)
# Remover headers/footers repetitivos comunes
text = re.sub(r'(?i)^.*medicina interna.*edici[oó]n.*$', '', text, flags=re.MULTILINE)
# Normalizar espacios
text = re.sub(r'[ \t]+', ' ', text)
# Trim líneas
text = '\n'.join(line.strip() for line in text.split('\n'))
return text.strip()
def generate_chunks(pages: list, libro_id: str, target_size: int = 800) -> list:
"""Genera chunks semánticos a partir de páginas.
Intenta cortar en límites de sección/párrafo, no a mitad de oración.
"""
chunks = []
current_chunk = ""
current_start_page = None
for page_data in pages:
page_num = page_data["page"]
text = page_data["text"]
if current_start_page is None:
current_start_page = page_num
paragraphs = text.split('\n\n')
for para in paragraphs:
para = para.strip()
if not para or len(para) < 10:
continue
word_count = len(current_chunk.split())
para_words = len(para.split())
# Si agregar este párrafo excede el target, guardar chunk actual
if word_count + para_words > target_size and word_count > 100:
chunks.append({
"id": f"{libro_id}_chunk_{len(chunks):05d}",
"libro_id": libro_id,
"page_start": current_start_page,
"page_end": page_num,
"text": current_chunk.strip(),
"word_count": word_count
})
current_chunk = para + "\n\n"
current_start_page = page_num
else:
current_chunk += para + "\n\n"
# Último chunk
if current_chunk.strip() and len(current_chunk.split()) > 50:
chunks.append({
"id": f"{libro_id}_chunk_{len(chunks):05d}",
"libro_id": libro_id,
"page_start": current_start_page,
"page_end": pages[-1]["page"] if pages else 0,
"text": current_chunk.strip(),
"word_count": len(current_chunk.split())
})
return chunks
def parse_libro(libro_id: str):
"""Parsea un libro por su ID del catálogo."""
catalog = load_catalog()
libro = None
for lib in catalog["libros"]:
if lib["id"] == libro_id:
libro = lib
break
if not libro:
print(f"Libro '{libro_id}' no encontrado en catalogo.")
return
if libro["tipo_pdf"] == "escaneado":
print(f"'{libro['titulo']}' es un PDF escaneado. Requiere OCR. Saltando.")
return
if libro["estado"] == "parseado":
print(f"'{libro['titulo']}' ya fue parseado. Usa --force para re-parsear.")
return
pdf_path = os.path.join(LIBROS_DIR, libro["archivo"])
if not os.path.exists(pdf_path):
print(f"Archivo no encontrado: {pdf_path}")
return
# Parsear
result = parse_pdf(pdf_path, libro_id)
# Guardar resultado (sin pages_data para ahorrar espacio, solo chunks)
os.makedirs(OUTPUT_DIR, exist_ok=True)
# Guardar chunks
chunks_path = os.path.join(OUTPUT_DIR, f"{libro_id}_chunks.json")
with open(chunks_path, "w", encoding="utf-8") as f:
json.dump(result["chunks_data"], f, ensure_ascii=False, indent=2)
# Guardar metadata
meta_path = os.path.join(OUTPUT_DIR, f"{libro_id}_meta.json")
meta = {k: v for k, v in result.items() if k not in ("pages_data", "chunks_data")}
with open(meta_path, "w", encoding="utf-8") as f:
json.dump(meta, f, ensure_ascii=False, indent=2)
# Actualizar catálogo
libro["estado"] = "parseado"
libro["fecha_parseo"] = result["fecha_parseo"]
libro["chunks_generados"] = result["chunks"]
save_catalog(catalog)
print(f" Guardado en: {chunks_path}")
print(f" Catalogo actualizado.")
def parse_all():
"""Parsea todos los libros digitales pendientes."""
catalog = load_catalog()
for libro in catalog["libros"]:
if libro["tipo_pdf"] == "digital" and libro["estado"] == "pendiente":
print(f"\n{'='*60}")
print(f" {libro['titulo']}")
print(f"{'='*60}")
parse_libro(libro["id"])
elif libro["tipo_pdf"] == "escaneado":
print(f"\n SKIP: {libro['titulo']} (escaneado, requiere OCR)")
elif libro["estado"] == "parseado":
print(f"\n SKIP: {libro['titulo']} (ya parseado)")
def status():
"""Muestra estado del parseo."""
catalog = load_catalog()
print(f"\n{'='*60}")
print(f" ESTADO DE PARSEO")
print(f"{'='*60}")
for libro in catalog["libros"]:
status_icon = {
"pendiente": "[ ]",
"parseado": "[x]",
"requiere_ocr": "[!]"
}.get(libro["estado"], "[?]")
chunks_info = f" ({libro['chunks_generados']} chunks)" if libro["chunks_generados"] else ""
print(f" {status_icon} {libro['titulo']} - {libro['paginas']} pags - {libro['tipo_pdf']}{chunks_info}")
if catalog.get("libros_faltantes"):
print(f"\n Libros no disponibles aun:")
for lib in catalog["libros_faltantes"]:
print(f" [-] {lib}")
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print("\nUso:")
print(" python parser.py status - Ver estado")
print(" python parser.py all - Parsear todos los pendientes")
print(" python parser.py <libro_id> - Parsear un libro especifico")
print("\nLibros disponibles:")
catalog = load_catalog()
for lib in catalog["libros"]:
print(f" {lib['id']}: {lib['titulo']} ({lib['estado']})")
sys.exit(0)
cmd = sys.argv[1]
if cmd == "status":
status()
elif cmd == "all":
parse_all()
elif cmd == "--force" and len(sys.argv) > 2:
libro_id = sys.argv[2]
catalog = load_catalog()
for lib in catalog["libros"]:
if lib["id"] == libro_id:
lib["estado"] = "pendiente"
save_catalog(catalog)
parse_libro(libro_id)
else:
parse_libro(cmd)

573
parser_v2.py Normal file
View File

@ -0,0 +1,573 @@
"""Parser v2: PDF → chunks semánticos con estructura, overlap y parent-child.
Módulo importable. Funciones principales:
- parse_libro_v2(libro_id, pdf_path) → (children, parents)
- detect_structure(pages, libro_id) → structured_pages
- generate_chunks_v2(structured_pages, libro_id) → (children, parents)
CLI: python parser_v2.py <libro_id> [--all] [--status]
"""
import fitz # PyMuPDF
import json
import os
import re
import unicodedata
from datetime import datetime
LIBROS_DIR = os.path.join(os.path.dirname(__file__), "..", "LIBROS")
OUTPUT_DIR = os.path.join(os.path.dirname(__file__), "parsed")
CATALOG_PATH = os.path.join(os.path.dirname(__file__), "catalog.json")
# --- Configuración de chunking ---
TARGET_SIZE = 280 # palabras objetivo por child chunk
MIN_SIZE = 150 # mínimo aceptable
MAX_SIZE = 380 # máximo antes de forzar corte
OVERLAP_SIZE = 60 # palabras de overlap entre chunks
PARENT_WINDOW = 3 # cantidad de children por parent chunk
MAX_PARENT_WORDS = 1200 # máximo de palabras por parent
# --- Patrones de estructura por libro ---
STRUCTURE_PATTERNS = {
"farreras-2020": {
"capitulo": [
r'^SECCIÓN\s+[IVXLCDM]+\b',
r'^Capítulo\s+\d+',
r'^CAPÍTULO\s+\d+',
],
"seccion": [
r'^\d+\.\d+[\s\.]+[A-ZÁÉÍÓÚÑ]', # 23.4 Glaucoma...
r'^[A-ZÁÉÍÓÚÑ][A-ZÁÉÍÓÚÑ\s]{5,60}$', # MODELOS DE REGRESIÓN (línea sola en mayúsculas)
],
},
"garcia-feijoo-2012": {
"capitulo": [
r'PA\s*R\s*T\s*E\s+\d+', # PA R T E 1 : BÁSICO
r'^PARTE\s+\d+',
],
"seccion": [
r'^\d+\s*\|\s*.+', # 1 | Embriología. Desarrollo...
r'^\d+\.\s+[A-ZÁÉÍÓÚÑ][a-záéíóúñ]{3,}', # 1. Embriología... (requiere palabra real, no "3. Mixto.")
],
},
"diamante-orl": {
"capitulo": [
r'^SECCIÓN\s+[IVXLCDM]+',
r'^Sección\s+[IVXLCDM]+',
r'^CAPÍTULO\s+\d+',
],
"seccion": [
r'^\d+\.\s+[A-ZÁÉÍÓÚÑ]',
r'^[A-ZÁÉÍÓÚÑ][A-ZÁÉÍÓÚÑ\s]{5,50}$',
],
},
"_default": {
"capitulo": [
r'^SECCIÓN\s+[IVXLCDM]+',
r'^CAPÍTULO\s+\d+',
r'^Capítulo\s+\d+',
r'^PARTE\s+\d+',
],
"seccion": [
r'^\d+\.\d+[\s\.]+[A-ZÁÉÍÓÚÑ]',
r'^[A-ZÁÉÍÓÚÑ][A-ZÁÉÍÓÚÑ\s]{5,60}$',
],
},
}
def load_catalog():
with open(CATALOG_PATH, "r", encoding="utf-8") as f:
return json.load(f)
def save_catalog(catalog):
with open(CATALOG_PATH, "w", encoding="utf-8") as f:
json.dump(catalog, f, ensure_ascii=False, indent=2)
def normalize_for_search(text: str) -> str:
"""Quita acentos y pasa a minúsculas para full-text search."""
nfkd = unicodedata.normalize('NFKD', text)
return ''.join(c for c in nfkd if not unicodedata.combining(c)).lower()
def clean_text(text: str) -> str:
"""Limpia texto extraído de PDF."""
text = re.sub(r'\n{3,}', '\n\n', text)
text = re.sub(r'^\s*\d{1,4}\s*$', '', text, flags=re.MULTILINE)
text = re.sub(r'(?i)^.*medicina interna.*edici[oó]n.*$', '', text, flags=re.MULTILINE)
text = re.sub(r'(?i)^.*booksmedicos\.org.*$', '', text, flags=re.MULTILINE)
text = re.sub(r'(?i)^.*© Elsevier\. Fotocopiar sin autorización es un delito\..*$', '', text, flags=re.MULTILINE)
text = re.sub(r'(?i)^.*© \d{4}.*Elsevier.*$', '', text, flags=re.MULTILINE)
text = re.sub(r'[ \t]+', ' ', text)
text = '\n'.join(line.strip() for line in text.split('\n'))
return text.strip()
def classify_content_type(text: str) -> str:
"""Clasifica el tipo de contenido de un chunk."""
lines = text.strip().split('\n')
if not lines:
return "body"
# Tablas: muchas líneas con | o tabulaciones
tab_lines = sum(1 for l in lines if '|' in l or '\t' in l)
if tab_lines > len(lines) * 0.3 and tab_lines >= 3:
return "tabla"
# Listas: muchas líneas empezando con -, *, •, números
list_lines = sum(1 for l in lines if re.match(r'^\s*[-*•●■]\s', l) or re.match(r'^\s*\d+[\.\)]\s', l))
if list_lines > len(lines) * 0.3 and list_lines >= 3:
return "lista"
# Definiciones: empieza con patrones típicos
first_100 = text[:200].lower()
if any(p in first_100 for p in ['concepto', 'definición', 'se define como', 'se denomina', 'es la']):
return "definicion"
return "body"
def _clean_spaced_text(text: str) -> str:
"""Reconstruye texto con letras espaciadas del PDF.
"PA R T E 3 : A M E T R O P Í A S" → "PARTE 3 : AMETROPÍAS"
"B Á S I C O" → "BÁSICO"
"""
# Buscar secuencias donde hay letras sueltas separadas por un espacio
# Patrón: al menos 2 pares de "letra espacio" seguidos de una letra final
letter = r'[A-ZÁÉÍÓÚÑa-záéíóúñ]'
pattern = rf'({letter} ){{2,}}{letter}'
def collapse_spaced(match):
return match.group(0).replace(' ', '')
result = re.sub(pattern, collapse_spaced, text)
return result
def detect_structure(pages: list, libro_id: str) -> list:
"""Detecta títulos de capítulo y sección en las páginas.
Args:
pages: Lista de {page, text}
libro_id: ID del libro para seleccionar patrones
Returns:
Lista de {page, text, titulo_capitulo, titulo_seccion}
"""
patterns = STRUCTURE_PATTERNS.get(libro_id, STRUCTURE_PATTERNS["_default"])
cap_patterns = [re.compile(p, re.MULTILINE) for p in patterns["capitulo"]]
sec_patterns = [re.compile(p, re.MULTILINE) for p in patterns["seccion"]]
current_capitulo = ""
current_seccion = ""
structured = []
for page_data in pages:
text = page_data["text"]
lines = text.split('\n')
for line in lines:
line_stripped = line.strip()
if not line_stripped or len(line_stripped) < 3:
continue
# Detectar capítulo
for pat in cap_patterns:
if pat.match(line_stripped):
# Limpiar: reconstruir texto con espacios intercalados
# "PA R T E 3 : A M E T R O P Í A S" → "PARTE 3: AMETROPÍAS"
clean_cap = _clean_spaced_text(line_stripped)
clean_cap = re.sub(r'\s{2,}', ' ', clean_cap).strip()
current_capitulo = clean_cap[:120]
current_seccion = ""
break
# Detectar sección (solo si no fue detectado como capítulo)
is_capitulo = any(pat.match(line_stripped) for pat in cap_patterns)
if not is_capitulo:
for pat in sec_patterns:
if pat.match(line_stripped):
# Solo aceptar como sección si es línea corta (título, no contenido)
# y no contiene punto seguido de más texto (indica oración, no título)
if len(line_stripped) < 80 and line_stripped.count('.') <= 2:
current_seccion = line_stripped[:100]
break
structured.append({
"page": page_data["page"],
"text": text,
"titulo_capitulo": current_capitulo,
"titulo_seccion": current_seccion,
})
return structured
def _find_sentence_boundary(words: list, target_idx: int) -> int:
"""Busca el final de oración más cercano al target_idx.
Retorna el índice del último word que termina una oración,
buscando en un rango de ±30 palabras del target.
"""
search_start = max(0, target_idx - 30)
search_end = min(len(words), target_idx + 30)
best = target_idx
best_dist = 999
for i in range(search_start, search_end):
word = words[i]
if word.endswith(('.', '?', '!', ':')) and not re.match(r'^\d+\.$', word):
dist = abs(i - target_idx)
if dist < best_dist:
best = i
best_dist = dist
return best + 1 # Retorna posición después del punto
def generate_chunks_v2(structured_pages: list, libro_id: str,
target_size: int = TARGET_SIZE,
overlap: int = OVERLAP_SIZE) -> tuple:
"""Genera child chunks y parent chunks a partir de páginas estructuradas.
Args:
structured_pages: Lista de {page, text, titulo_capitulo, titulo_seccion}
libro_id: ID del libro
target_size: Palabras objetivo por chunk (default 280)
overlap: Palabras de overlap (default 60)
Returns:
(child_chunks, parent_chunks)
"""
# Construir un buffer continuo con metadata por página
all_words = [] # Lista plana de palabras
word_meta = [] # Metadata por palabra: (page, capitulo, seccion)
for sp in structured_pages:
page = sp["page"]
cap = sp["titulo_capitulo"]
sec = sp["titulo_seccion"]
words = sp["text"].split()
for w in words:
all_words.append(w)
word_meta.append((page, cap, sec))
if not all_words:
return [], []
# Generar child chunks con overlap
children = []
pos = 0
chunk_index = 0
while pos < len(all_words):
# Determinar fin del chunk
end_target = pos + target_size
if end_target >= len(all_words):
# Último chunk: tomar todo lo que queda
end = len(all_words)
else:
# Buscar límite de oración cerca del target
end = _find_sentence_boundary(all_words, end_target)
# Si el chunk es demasiado grande, forzar corte
if end - pos > MAX_SIZE:
end = _find_sentence_boundary(all_words, pos + MAX_SIZE)
if end - pos > MAX_SIZE + 50:
end = pos + MAX_SIZE # Corte duro como último recurso
# Chunk demasiado pequeño al final: merge con anterior
if end - pos < MIN_SIZE and children:
prev = children[-1]
prev["text"] = prev["text"] + " " + " ".join(all_words[pos:end])
prev["word_count"] = len(prev["text"].split())
prev["page_end"] = word_meta[end - 1][0]
break
chunk_text = " ".join(all_words[pos:end])
page_start = word_meta[pos][0]
page_end = word_meta[end - 1][0]
# Metadata: tomar la del inicio del chunk (más representativa)
capitulo = word_meta[pos][1]
seccion = word_meta[pos][2]
# Si hay cambio de capítulo/sección dentro del chunk, usar el más nuevo
for i in range(pos, min(end, pos + 50)):
if word_meta[i][1] and word_meta[i][1] != capitulo:
capitulo = word_meta[i][1]
if word_meta[i][2] and word_meta[i][2] != seccion:
seccion = word_meta[i][2]
tipo = classify_content_type(chunk_text)
children.append({
"id": f"{libro_id}_v2_{chunk_index:05d}",
"libro_id": libro_id,
"page_start": page_start,
"page_end": page_end,
"text": chunk_text,
"word_count": end - pos,
"titulo_capitulo": capitulo,
"titulo_seccion": seccion,
"tipo_contenido": tipo,
"parent_id": None, # Se asigna después
"chunk_index": chunk_index,
"version": 2,
})
chunk_index += 1
# Avanzar con overlap
next_pos = end - overlap
if next_pos <= pos:
next_pos = end # Evitar loop infinito
pos = next_pos
# Generar parent chunks (ventana de PARENT_WINDOW children)
parents = []
parent_idx = 0
i = 0
while i < len(children):
window_end = min(i + PARENT_WINDOW, len(children))
window = children[i:window_end]
# Concatenar textos de los children (sin overlap duplicado)
parent_text_parts = []
for j, child in enumerate(window):
if j == 0:
parent_text_parts.append(child["text"])
else:
# Quitar overlap del inicio de este child (ya está en el anterior)
child_words = child["text"].split()
# El overlap son las últimas ~OVERLAP_SIZE palabras del child anterior
skip = min(overlap, len(child_words) // 3) # No skipear más de 1/3
parent_text_parts.append(" ".join(child_words[skip:]))
parent_text = " ".join(parent_text_parts)
parent_words = len(parent_text.split())
# Si el parent es muy grande, solo tomar lo necesario
if parent_words > MAX_PARENT_WORDS:
parent_text = " ".join(parent_text.split()[:MAX_PARENT_WORDS])
parent_words = MAX_PARENT_WORDS
parent_id = f"{libro_id}_v2_parent_{parent_idx:05d}"
parents.append({
"id": parent_id,
"libro_id": libro_id,
"page_start": window[0]["page_start"],
"page_end": window[-1]["page_end"],
"text": parent_text,
"word_count": parent_words,
"titulo_capitulo": window[0]["titulo_capitulo"],
"titulo_seccion": window[0]["titulo_seccion"],
"child_ids": [c["id"] for c in window],
"version": 2,
})
# Asignar parent_id a los children
for child in window:
child["parent_id"] = parent_id
parent_idx += 1
i += PARENT_WINDOW
return children, parents
def parse_pdf_v2(pdf_path: str, libro_id: str) -> tuple:
"""Parsea un PDF completo a chunks v2.
Args:
pdf_path: Ruta al archivo PDF
libro_id: ID del libro
Returns:
(children, parents) — listas de dicts
"""
doc = fitz.open(pdf_path)
total_pages = doc.page_count
print(f" Parseando {os.path.basename(pdf_path)} ({total_pages} págs)...")
# Extraer texto por página
pages = []
for i in range(total_pages):
page = doc[i]
text = page.get_text()
text = clean_text(text)
if len(text.strip()) > 20:
pages.append({"page": i + 1, "text": text})
if (i + 1) % 500 == 0:
print(f" ... {i + 1}/{total_pages} páginas")
doc.close()
print(f" {len(pages)} páginas con texto extraído")
# Detectar estructura
structured = detect_structure(pages, libro_id)
# Generar chunks
children, parents = generate_chunks_v2(structured, libro_id)
print(f" Resultado: {len(children)} children, {len(parents)} parents")
return children, parents
def parse_libro_v2(libro_id: str, pdf_path: str = None,
on_progress: callable = None) -> tuple:
"""Parsea un libro por ID o ruta directa. Función principal importable.
Args:
libro_id: ID del libro en el catálogo
pdf_path: Ruta al PDF (opcional, se busca en catálogo si no se da)
on_progress: Callback(step, pct, msg) para reportar progreso
Returns:
(children, parents) — listas de dicts
"""
def report(pct, msg):
if on_progress:
on_progress("parse", pct, msg)
print(f" [{pct}%] {msg}")
# Resolver ruta del PDF
if not pdf_path:
catalog = load_catalog()
libro = next((l for l in catalog["libros"] if l["id"] == libro_id), None)
if not libro:
raise ValueError(f"Libro '{libro_id}' no encontrado en catálogo")
pdf_path = os.path.join(LIBROS_DIR, libro["archivo"])
if not os.path.exists(pdf_path):
raise FileNotFoundError(f"PDF no encontrado: {pdf_path}")
report(0, f"Iniciando parseo de {libro_id}")
# Parsear
children, parents = parse_pdf_v2(pdf_path, libro_id)
# Guardar JSONs
os.makedirs(OUTPUT_DIR, exist_ok=True)
children_path = os.path.join(OUTPUT_DIR, f"{libro_id}_v2_chunks.json")
with open(children_path, "w", encoding="utf-8") as f:
json.dump(children, f, ensure_ascii=False, indent=2)
parents_path = os.path.join(OUTPUT_DIR, f"{libro_id}_v2_parents.json")
with open(parents_path, "w", encoding="utf-8") as f:
json.dump(parents, f, ensure_ascii=False, indent=2)
report(100, f"Guardado: {len(children)} children -> {children_path}")
# Actualizar catálogo
catalog = load_catalog()
libro_entry = next((l for l in catalog["libros"] if l["id"] == libro_id), None)
if libro_entry:
libro_entry["chunks_v2"] = len(children)
libro_entry["parents_v2"] = len(parents)
libro_entry["fecha_parseo_v2"] = datetime.now().isoformat()
save_catalog(catalog)
return children, parents
# --- CLI ---
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print("\nparser_v2 — Chunking semántico con estructura, overlap y parent-child")
print("\nUso:")
print(" python parser_v2.py status - Ver estado")
print(" python parser_v2.py <libro_id> - Parsear un libro")
print(" python parser_v2.py all - Parsear todos los digitales")
print(" python parser_v2.py <libro_id> --preview - Preview: 10 chunks sin guardar")
catalog = load_catalog()
print("\nLibros disponibles:")
for lib in catalog["libros"]:
v2 = f" (v2: {lib.get('chunks_v2', 0)} chunks)" if lib.get('chunks_v2') else ""
print(f" {lib['id']}: {lib['titulo']} ({lib['estado']}){v2}")
sys.exit(0)
cmd = sys.argv[1]
if cmd == "status":
catalog = load_catalog()
print(f"\n{'='*60}")
print(f" ESTADO DE PARSEO V2")
print(f"{'='*60}")
for lib in catalog["libros"]:
v2_chunks = lib.get("chunks_v2", 0)
v2_parents = lib.get("parents_v2", 0)
if v2_chunks:
icon = "[v2]"
info = f" → {v2_chunks} children, {v2_parents} parents"
elif lib["estado"] == "parseado":
icon = "[v1]"
info = f" → {lib.get('chunks_generados', 0)} chunks v1"
else:
icon = "[ ]"
info = ""
print(f" {icon} {lib['titulo']} ({lib['paginas']} págs, {lib['tipo_pdf']}){info}")
elif cmd == "all":
catalog = load_catalog()
for lib in catalog["libros"]:
if lib["tipo_pdf"] == "digital" and lib["estado"] == "parseado":
print(f"\n{'='*60}")
print(f" {lib['titulo']}")
print(f"{'='*60}")
parse_libro_v2(lib["id"])
elif lib["tipo_pdf"] == "escaneado":
print(f"\n SKIP: {lib['titulo']} (escaneado)")
elif "--preview" in sys.argv:
libro_id = cmd
catalog = load_catalog()
libro = next((l for l in catalog["libros"] if l["id"] == libro_id), None)
if not libro:
print(f"Libro '{libro_id}' no encontrado")
sys.exit(1)
pdf_path = os.path.join(LIBROS_DIR, libro["archivo"])
children, parents = parse_pdf_v2(pdf_path, libro_id)
print(f"\n{'='*60}")
print(f" PREVIEW: {len(children)} children, {len(parents)} parents")
print(f"{'='*60}")
# Mostrar stats
sizes = [c["word_count"] for c in children]
print(f" Tamaño promedio: {sum(sizes)/len(sizes):.0f} palabras")
print(f" Min: {min(sizes)}, Max: {max(sizes)}")
caps = set(c["titulo_capitulo"] for c in children if c["titulo_capitulo"])
secs = set(c["titulo_seccion"] for c in children if c["titulo_seccion"])
print(f" Capítulos detectados: {len(caps)}")
print(f" Secciones detectadas: {len(secs)}")
# Mostrar 10 chunks del medio (no prólogos)
start = len(children) // 3
for c in children[start:start+10]:
print(f"\n--- [{c['id']}] págs {c['page_start']}-{c['page_end']} ({c['word_count']} words) ---")
print(f" Cap: {c['titulo_capitulo'][:60] if c['titulo_capitulo'] else '(no detectado)'}")
print(f" Sec: {c['titulo_seccion'][:60] if c['titulo_seccion'] else '(no detectado)'}")
print(f" Tipo: {c['tipo_contenido']}")
print(f" Parent: {c['parent_id']}")
preview_text = c['text'][:200].encode('ascii', 'replace').decode('ascii')
print(f" Texto: {preview_text}...")
else:
parse_libro_v2(cmd)

259
quickstart.py Normal file
View File

@ -0,0 +1,259 @@
"""MedGraph Quickstart -- Get up and running in 5 minutes.
Place a PDF in the examples/ folder, configure .env, and run this script.
It will parse your PDF, generate embeddings, extract medical entities,
and verify the system works with a sample query.
Usage:
python quickstart.py
python quickstart.py --skip-extract # skip entity extraction (faster)
"""
import os
import sys
import glob
import time
def check_env():
"""Verify .env is configured."""
print("\n[1/7] Checking environment...")
if not os.path.exists(".env"):
print(" ERROR: .env file not found.")
print(" Run: cp .env.example .env")
print(" Then edit .env with your Neo4j and GCP credentials.")
return False
from dotenv import load_dotenv
load_dotenv()
required = ["NEO4J_URI", "NEO4J_USERNAME", "NEO4J_PASSWORD"]
missing = [v for v in required if not os.getenv(v)]
if missing:
print(f" ERROR: Missing environment variables: {', '.join(missing)}")
print(" Edit your .env file with the correct values.")
return False
print(" OK - Environment configured")
return True
def check_dependencies():
"""Verify Python dependencies are installed."""
print("\n[2/7] Checking dependencies...")
deps = {
"neo4j": "neo4j",
"fitz": "PyMuPDF (pip install pymupdf)",
"dotenv": "python-dotenv",
}
missing = []
for module, name in deps.items():
try:
__import__(module)
except ImportError:
missing.append(name)
if missing:
print(f" ERROR: Missing packages: {', '.join(missing)}")
print(" Run: pip install -r requirements.txt")
return False
print(" OK - All dependencies installed")
return True
def setup_schema():
"""Create Neo4j schema and indexes."""
print("\n[3/7] Setting up Neo4j schema...")
try:
from schema import create_schema
create_schema()
print(" OK - Schema and indexes created")
return True
except ImportError:
print(" WARN: schema.py not found, skipping schema creation")
return True
except Exception as e:
print(f" ERROR: {e}")
print(" Check your Neo4j credentials in .env")
return False
def find_pdf():
"""Find a PDF in the examples folder."""
print("\n[4/7] Looking for PDFs in examples/...")
pdfs = glob.glob("examples/*.pdf")
if not pdfs:
print(" ERROR: No PDF files found in examples/")
print(" Place a PDF in the examples/ folder and try again.")
print(" See examples/README.md for suggestions on free medical PDFs.")
return None
pdf = pdfs[0]
print(f" Found: {pdf}")
if len(pdfs) > 1:
print(f" (Using first one. {len(pdfs)} PDFs found total)")
return pdf
def run_pipeline(pdf_path, skip_extract=False):
"""Run the full MedGraph pipeline on a PDF."""
# Derive libro_id from filename
libro_id = os.path.splitext(os.path.basename(pdf_path))[0]
libro_id = libro_id.lower().replace(" ", "-").replace("_", "-")
print(f"\n Book ID: {libro_id}")
# Step 1: Parse
print("\n[5/7] Parsing PDF...")
try:
from parser_v2 import parse_libro_v2
children, parents = parse_libro_v2(libro_id, pdf_path)
print(f" OK - {len(children)} chunks, {len(parents)} parent chunks")
except Exception as e:
print(f" ERROR parsing: {e}")
return False
if not children:
print(" ERROR: No chunks generated. The PDF might be scanned (needs OCR).")
return False
# Step 2: Upload to Neo4j
print("\n[6/7] Uploading to Neo4j and generating embeddings...")
try:
from migrate_chunks import upload_chunks_for_libro, normalize_chunks
normalize_chunks(children)
stats = upload_chunks_for_libro(libro_id, children, parents)
print(f" Uploaded: {stats}")
except ImportError:
try:
from upload_chunks import upload_chunks
upload_chunks(libro_id, children, parents)
print(f" Uploaded: {len(children)} chunks")
except Exception as e:
print(f" ERROR uploading: {e}")
return False
except Exception as e:
print(f" ERROR uploading: {e}")
return False
# Step 3: Vectorize
try:
from vectorize import vectorize_libro
vectorize_libro(libro_id)
print(f" OK - Embeddings generated")
except Exception as e:
print(f" WARN: Vectorization failed: {e}")
print(" You may need to configure GCP_API_KEY in .env")
# Step 4: Extract entities (optional)
if skip_extract:
print("\n[7/7] Skipping entity extraction (--skip-extract)")
else:
print("\n[7/7] Extracting medical entities (this may take a few minutes)...")
try:
from extract_entities import init_model, extract_from_chunk, canonicalize_entities, upload_entities
import json
model = init_model()
all_ext = []
for i, chunk in enumerate(children):
ext = extract_from_chunk(model, chunk, libro_id)
all_ext.append(ext)
if (i + 1) % 10 == 0:
pct = int((i + 1) / len(children) * 100)
print(f" [{pct}%] {i + 1}/{len(children)} chunks processed")
if (i + 1) % 5 == 0:
time.sleep(1) # Rate limiting
entities, relations = canonicalize_entities(all_ext)
print(f" Extracted: {len(entities)} entities, {len(relations)} relationships")
# Save extraction
os.makedirs("extracted", exist_ok=True)
with open(f"extracted/{libro_id}_entities.json", "w", encoding="utf-8") as f:
json.dump({"extractions": all_ext}, f, ensure_ascii=False)
upload_entities(entities, relations, libro_id, dev_mode=False)
print(f" OK - Entities uploaded to Neo4j")
except Exception as e:
print(f" WARN: Entity extraction failed: {e}")
print(" The system still works for bibliography search without entities.")
return True
def verify():
"""Run a sample query to verify the system works."""
print("\n--- Verification ---")
try:
from db import run_query
chunks = run_query("MATCH (c:Chunk) RETURN count(c) AS n")[0]["n"]
entities = run_query("MATCH (n) WHERE NOT n:Chunk AND NOT n:ParentChunk RETURN count(n) AS n")[0]["n"]
rels = run_query("MATCH ()-[r]->() RETURN count(r) AS n")[0]["n"]
print(f"\n Your MedGraph instance:")
print(f" Chunks: {chunks:,}")
print(f" Entities: {entities:,}")
print(f" Relationships:{rels:,}")
if chunks > 0:
print("\n SUCCESS! MedGraph is ready.")
print(f"\n Next steps:")
print(f" - Start the API: cd api && uvicorn main:app --reload")
print(f" - Add more books: python quickstart.py (with new PDFs in examples/)")
print(f" - Run ontology: python ontology.py")
return True
else:
print("\n WARNING: No chunks found. Something went wrong in the pipeline.")
return False
except Exception as e:
print(f" ERROR verifying: {e}")
return False
def main():
print("=" * 50)
print(" MedGraph Quickstart")
print("=" * 50)
skip_extract = "--skip-extract" in sys.argv
if not check_env():
sys.exit(1)
if not check_dependencies():
sys.exit(1)
if not setup_schema():
sys.exit(1)
pdf = find_pdf()
if not pdf:
sys.exit(1)
if not run_pipeline(pdf, skip_extract=skip_extract):
sys.exit(1)
verify()
print("\n" + "=" * 50)
print(" Done!")
print("=" * 50 + "\n")
if __name__ == "__main__":
main()

3
requirements.txt Normal file
View File

@ -0,0 +1,3 @@
neo4j>=5.0
pymupdf>=1.24
python-dotenv>=1.0

107
schema.py Normal file
View File

@ -0,0 +1,107 @@
"""Define y crea el schema de MedGraph en Neo4j."""
from db import run_write, run_query
CONSTRAINTS = [
# Unicidad por nombre para cada tipo de nodo
"CREATE CONSTRAINT patologia_nombre IF NOT EXISTS FOR (n:Patologia) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT sintoma_nombre IF NOT EXISTS FOR (n:Sintoma) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT signo_nombre IF NOT EXISTS FOR (n:Signo) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT metodo_dx_nombre IF NOT EXISTS FOR (n:MetodoDx) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT tratamiento_nombre IF NOT EXISTS FOR (n:Tratamiento) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT farmaco_nombre IF NOT EXISTS FOR (n:Farmaco) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT procedimiento_nombre IF NOT EXISTS FOR (n:Procedimiento) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT parametro_nombre IF NOT EXISTS FOR (n:Parametro) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT up_id IF NOT EXISTS FOR (n:UP) REQUIRE n.id IS UNIQUE",
"CREATE CONSTRAINT tema_nombre IF NOT EXISTS FOR (n:Tema) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT fuente_id IF NOT EXISTS FOR (n:Fuente) REQUIRE n.id IS UNIQUE",
"CREATE CONSTRAINT especialidad_nombre IF NOT EXISTS FOR (n:Especialidad) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT agente_nombre IF NOT EXISTS FOR (n:Agente) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT grupo_farmacologico_nombre IF NOT EXISTS FOR (n:GrupoFarmacologico) REQUIRE n.nombre IS UNIQUE",
"CREATE CONSTRAINT grupo_etario_nombre IF NOT EXISTS FOR (n:GrupoEtario) REQUIRE n.nombre IS UNIQUE",
# v2: chunks y parent chunks
"CREATE CONSTRAINT chunk_id IF NOT EXISTS FOR (c:Chunk) REQUIRE c.id IS UNIQUE",
"CREATE CONSTRAINT parent_chunk_id IF NOT EXISTS FOR (p:ParentChunk) REQUIRE p.id IS UNIQUE",
]
INDEXES = [
# Índices full-text para búsqueda por texto
"""CREATE FULLTEXT INDEX busqueda_patologias IF NOT EXISTS
FOR (n:Patologia) ON EACH [n.nombre, n.definicion]""",
"""CREATE FULLTEXT INDEX busqueda_sintomas IF NOT EXISTS
FOR (n:Sintoma) ON EACH [n.nombre, n.descripcion]""",
"""CREATE FULLTEXT INDEX busqueda_temas IF NOT EXISTS
FOR (n:Tema) ON EACH [n.nombre, n.descripcion]""",
# Full-text index sobre chunks v2 (normalizado, multi-campo)
"""CREATE FULLTEXT INDEX busqueda_chunks_v2 IF NOT EXISTS
FOR (n:Chunk) ON EACH [n.text_busqueda, n.titulo_seccion_busqueda,
n.titulo_capitulo_busqueda, n.keywords]""",
]
# Índices a borrar durante migración v2
INDEXES_TO_DROP = [
"DROP INDEX busqueda_chunks IF EXISTS",
]
def create_schema(drop_old: bool = False):
"""Crea constraints e índices en Neo4j.
Args:
drop_old: Si True, borra índices viejos (para migración v2)
"""
if drop_old:
print("Borrando índices viejos...")
for idx in INDEXES_TO_DROP:
try:
run_write(idx)
print(f" DROPPED: {idx}")
except Exception as e:
print(f" SKIP: {e}")
print("Creando constraints...")
for c in CONSTRAINTS:
try:
run_write(c)
# Extraer label del constraint
if "FOR (n:" in c:
label = c.split("FOR (n:")[1].split(")")[0]
elif "FOR (c:" in c:
label = c.split("FOR (c:")[1].split(")")[0]
elif "FOR (p:" in c:
label = c.split("FOR (p:")[1].split(")")[0]
else:
label = "?"
print(f" OK: {label}")
except Exception as e:
print(f" SKIP: {e}")
print("\nCreando índices full-text...")
for idx in INDEXES:
try:
run_write(idx)
name = idx.split("INDEX ")[1].split(" IF")[0]
print(f" OK: {name}")
except Exception as e:
print(f" SKIP: {e}")
print("\nSchema creado.")
def show_schema():
"""Muestra el schema actual."""
constraints = run_query("SHOW CONSTRAINTS")
indexes = run_query("SHOW INDEXES")
print(f"\n=== CONSTRAINTS ({len(constraints)}) ===")
for c in constraints:
print(f" {c.get('name', '?')}: {c.get('labelsOrTypes', '?')}")
print(f"\n=== INDEXES ({len(indexes)}) ===")
for i in indexes:
print(f" {i.get('name', '?')}: {i.get('labelsOrTypes', '?')} ({i.get('type', '?')})")
if __name__ == "__main__":
create_schema()
show_schema()

228
search.py Normal file
View File

@ -0,0 +1,228 @@
"""Búsqueda por keywords sobre los chunks parseados de los libros."""
import json
import os
import re
import sys
from collections import defaultdict
PARSED_DIR = os.path.join(os.path.dirname(__file__), "parsed")
CATALOG_PATH = os.path.join(os.path.dirname(__file__), "catalog.json")
# Cache de chunks cargados en memoria
_chunks_cache = {}
def load_chunks(libro_id: str = None) -> list:
"""Carga chunks de uno o todos los libros parseados."""
if libro_id and libro_id in _chunks_cache:
return _chunks_cache[libro_id]
catalog = load_catalog()
chunks = []
for libro in catalog["libros"]:
if libro["estado"] != "parseado":
continue
if libro_id and libro["id"] != libro_id:
continue
chunks_path = os.path.join(PARSED_DIR, f"{libro['id']}_chunks.json")
if not os.path.exists(chunks_path):
continue
with open(chunks_path, "r", encoding="utf-8") as f:
libro_chunks = json.load(f)
for chunk in libro_chunks:
chunk["libro_titulo"] = libro["titulo"]
chunks.extend(libro_chunks)
_chunks_cache[libro["id"]] = libro_chunks
return chunks
def load_catalog():
with open(CATALOG_PATH, "r", encoding="utf-8") as f:
return json.load(f)
def search(query: str, libro_id: str = None, top_k: int = 5, context: int = 0) -> list:
"""Búsqueda por keywords sobre chunks.
Scoring: cuenta ocurrencias de cada término de búsqueda en el chunk,
ponderado por si aparece en las primeras líneas (probable título/sección).
Args:
query: Texto a buscar
libro_id: Filtrar por libro específico (None = todos)
top_k: Cantidad de resultados
context: Líneas de contexto adicional a mostrar
Returns:
Lista de resultados ordenados por relevancia
"""
chunks = load_chunks(libro_id)
terms = normalize(query).split()
if not terms:
return []
results = []
for chunk in chunks:
text_normalized = normalize(chunk["text"])
text_lower = text_normalized.lower()
# Calcular score
score = 0
matched_terms = set()
for term in terms:
term_lower = term.lower()
# Contar ocurrencias
count = text_lower.count(term_lower)
if count > 0:
matched_terms.add(term)
score += count
# Bonus si aparece en las primeras 200 chars (probablemente título)
if term_lower in text_lower[:200]:
score += 3
# Solo incluir si matchea al menos la mitad de los términos
if len(matched_terms) >= max(1, len(terms) // 2):
# Bonus por matchear más términos distintos
score += len(matched_terms) * 2
results.append({
"chunk_id": chunk["id"],
"libro_id": chunk["libro_id"],
"libro_titulo": chunk.get("libro_titulo", chunk["libro_id"]),
"page_start": chunk["page_start"],
"page_end": chunk["page_end"],
"score": score,
"matched_terms": list(matched_terms),
"preview": get_preview(chunk["text"], terms),
"full_text": chunk["text"] if context > 0 else None,
"word_count": chunk["word_count"]
})
# Ordenar por score descendente
results.sort(key=lambda x: x["score"], reverse=True)
return results[:top_k]
def normalize(text: str) -> str:
"""Normaliza texto para búsqueda."""
# Remover acentos comunes
replacements = {
'á': 'a', 'é': 'e', 'í': 'i', 'ó': 'o', 'ú': 'u',
'ü': 'u', 'ñ': 'n',
'Á': 'A', 'É': 'E', 'Í': 'I', 'Ó': 'O', 'Ú': 'U',
'Ü': 'U', 'Ñ': 'N'
}
for old, new in replacements.items():
text = text.replace(old, new)
return text
def safe_print(text: str):
"""Print safe para Windows cp1252."""
try:
print(text)
except UnicodeEncodeError:
print(text.encode("ascii", errors="replace").decode("ascii"))
def get_preview(text: str, terms: list, max_len: int = 300) -> str:
"""Extrae un preview del chunk centrado en los términos encontrados."""
text_lower = normalize(text).lower()
terms_lower = [normalize(t).lower() for t in terms]
# Buscar la primera ocurrencia de cualquier término
best_pos = len(text)
for term in terms_lower:
pos = text_lower.find(term)
if pos != -1 and pos < best_pos:
best_pos = pos
# Extraer ventana alrededor de la primera ocurrencia
start = max(0, best_pos - 80)
end = min(len(text), start + max_len)
preview = text[start:end].strip()
if start > 0:
preview = "..." + preview
if end < len(text):
preview = preview + "..."
return preview
def print_results(results: list, verbose: bool = False):
"""Imprime resultados de búsqueda."""
if not results:
print("\n Sin resultados.")
return
print(f"\n {len(results)} resultado(s):\n")
for i, r in enumerate(results):
safe_print(f" [{i+1}] {r['libro_titulo']} - pags {r['page_start']}-{r['page_end']} (score: {r['score']})")
safe_print(f" Terminos: {', '.join(r['matched_terms'])}")
safe_print(f" {r['preview']}")
if verbose and r.get("full_text"):
safe_print(f"\n --- TEXTO COMPLETO ---")
for line in r["full_text"].split("\n"):
safe_print(f" {line}")
safe_print(f" --- FIN ---")
print()
if __name__ == "__main__":
args = sys.argv[1:]
if not args:
print("\nUso:")
print(' python search.py "otitis media tratamiento"')
print(' python search.py "ECG normal" --libro farreras-2020')
print(' python search.py "otalgia" --top 10')
print(' python search.py "ergometria indicaciones" --verbose')
sys.exit(0)
# Parsear argumentos
query_parts = []
libro_id = None
top_k = 5
verbose = False
i = 0
while i < len(args):
if args[i] == "--libro" and i + 1 < len(args):
libro_id = args[i + 1]
i += 2
elif args[i] == "--top" and i + 1 < len(args):
top_k = int(args[i + 1])
i += 2
elif args[i] == "--verbose":
verbose = True
i += 1
else:
query_parts.append(args[i])
i += 1
query = " ".join(query_parts)
if not query:
print("Error: query vacia")
sys.exit(1)
print(f"\n Buscando: \"{query}\"", end="")
if libro_id:
print(f" (en {libro_id})", end="")
print()
results = search(query, libro_id=libro_id, top_k=top_k, context=1 if verbose else 0)
print_results(results, verbose=verbose)

176
tests/test_basic.py Normal file
View File

@ -0,0 +1,176 @@
"""Basic tests for MedGraph Engine core components."""
import os
import sys
import json
import pytest
# Add parent dir to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
class TestParser:
"""Test PDF parser and chunking logic."""
def test_normalize_name(self):
from extract_entities import normalize_name
assert normalize_name(" Otitis Media Aguda ") == "otitis media aguda"
assert normalize_name("hipertensión...") == "hipertensión"
assert normalize_name(" múltiples espacios ") == "múltiples espacios"
assert normalize_name("") == ""
def test_strip_accents(self):
from dedup_entities import strip_accents
assert strip_accents("hipertensión") == "hipertension"
assert strip_accents("diagnóstico") == "diagnostico"
assert strip_accents("farmacología") == "farmacologia"
assert strip_accents("normal") == "normal"
def test_has_accents(self):
from dedup_entities import has_accents
assert has_accents("hipertensión") is True
assert has_accents("hipertension") is False
def test_pick_canonical(self):
from dedup_entities import pick_canonical
nodes = [
{"nombre": "hipertension", "freq": 10, "sinonimos": []},
{"nombre": "hipertensión", "freq": 5, "sinonimos": ["HTA"]},
]
canon = pick_canonical(nodes)
assert canon["nombre"] == "hipertensión" # prefers accented
def test_find_accent_groups(self):
from dedup_entities import find_accent_groups
entities = [
{"nombre": "hipertensión"},
{"nombre": "hipertension"},
{"nombre": "diabetes"},
]
groups = find_accent_groups(entities)
assert len(groups) == 1
assert "hipertension" in groups
assert len(groups["hipertension"]) == 2
class TestChunking:
"""Test chunking parameters and structure."""
def test_chunk_config_values(self):
from parser_v2 import TARGET_SIZE, MIN_SIZE, MAX_SIZE, OVERLAP_SIZE
assert TARGET_SIZE == 280
assert MIN_SIZE == 150
assert MAX_SIZE == 380
assert OVERLAP_SIZE == 60
assert MIN_SIZE < TARGET_SIZE < MAX_SIZE
def test_parent_config(self):
from parser_v2 import PARENT_WINDOW, MAX_PARENT_WORDS
assert PARENT_WINDOW == 3
assert MAX_PARENT_WORDS == 1200
class TestExtraction:
"""Test entity extraction validation logic."""
def test_validate_entity_types(self):
from extract_entities import ENTITY_TYPES
expected = {
"patologia", "estructura_anatomica", "procedimiento",
"farmaco", "grupo_farmacologico", "agente", "signo",
"sintoma", "metodo_dx", "hallazgo", "parametro",
}
assert ENTITY_TYPES == expected
def test_validate_relation_types(self):
from extract_entities import RELATION_TYPES
assert "CAUSADA_POR" in RELATION_TYPES
assert "SE_TRATA_CON" in RELATION_TYPES
assert "SE_DIAGNOSTICA_CON" in RELATION_TYPES
assert "ASOCIADA_A" in RELATION_TYPES
def test_type_to_label_mapping(self):
from extract_entities import TYPE_TO_LABEL
assert TYPE_TO_LABEL["patologia"] == "Patologia"
assert TYPE_TO_LABEL["farmaco"] == "Farmaco"
assert TYPE_TO_LABEL["estructura_anatomica"] == "EstructuraAnatomica"
def test_validate_extraction_filters_invalid(self):
from extract_entities import _validate_extraction
data = {
"entidades": [
{"nombre": "otitis media", "tipo": "patologia", "sinonimos": []},
{"nombre": "x", "tipo": "patologia", "sinonimos": []}, # too short
{"nombre": "valid", "tipo": "INVALID_TYPE", "sinonimos": []}, # bad type
{"nombre": "", "tipo": "patologia", "sinonimos": []}, # empty
],
"relaciones": []
}
result = _validate_extraction(data, {"id": "test", "libro_id": "test"})
assert len(result["entidades"]) == 1
assert result["entidades"][0]["nombre"] == "otitis media"
class TestCanonicalization:
"""Test entity canonicalization logic."""
def test_canonicalize_merges_duplicates(self):
from extract_entities import canonicalize_entities
extractions = [
{
"entidades": [
{"nombre": "otitis media", "tipo": "patologia", "sinonimos": ["OMA"]},
],
"relaciones": [],
"chunk_id": "c1",
},
{
"entidades": [
{"nombre": "otitis media", "tipo": "patologia", "sinonimos": []},
],
"relaciones": [],
"chunk_id": "c2",
},
]
entities, relations = canonicalize_entities(extractions)
# Should merge into 1 entity
otitis = [e for e in entities if e["nombre"] == "otitis media"]
assert len(otitis) == 1
assert otitis[0]["freq"] == 2
assert "OMA" in otitis[0]["sinonimos"]
class TestAPIConfig:
"""Test API configuration basics."""
def test_env_example_exists(self):
assert os.path.exists(
os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), ".env.example")
)
def test_requirements_no_vertexai(self):
req_path = os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "api", "requirements.txt"
)
with open(req_path) as f:
content = f.read()
assert "vertexai" not in content.lower()
assert "aiplatform" not in content.lower()
def test_docker_compose_exists(self):
assert os.path.exists(
os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "docker-compose.yml")
)
def test_docker_compose_no_real_credentials(self):
compose_path = os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "docker-compose.yml"
)
with open(compose_path) as f:
content = f.read()
assert "changeme" in content
assert "AIzaSy" not in content
if __name__ == "__main__":
pytest.main([__file__, "-v"])

81
upload_chunks.py Normal file
View File

@ -0,0 +1,81 @@
"""Sube todos los chunks a Neo4j SIN embeddings. Rápido, para que keyword search funcione."""
import json
import os
from db import run_write, run_query
PARSED_DIR = os.path.join(os.path.dirname(__file__), "parsed")
CATALOG_PATH = os.path.join(os.path.dirname(__file__), "catalog.json")
def upload_all():
with open(CATALOG_PATH, "r", encoding="utf-8") as f:
catalog = json.load(f)
total = 0
for libro in catalog["libros"]:
if libro["estado"] != "parseado":
continue
chunks_path = os.path.join(PARSED_DIR, f"{libro['id']}_chunks.json")
if not os.path.exists(chunks_path):
continue
with open(chunks_path, "r", encoding="utf-8") as f:
chunks = json.load(f)
# Check how many already exist
existing = run_query(
"MATCH (c:Chunk {libro_id: $lid}) RETURN count(c) AS n",
{"lid": libro["id"]}
)
existing_count = existing[0]["n"] if existing else 0
if existing_count >= len(chunks):
print(f" {libro['titulo']}: {existing_count} chunks ya en Neo4j (skip)")
continue
print(f" {libro['titulo']}: subiendo {len(chunks)} chunks ({existing_count} ya existen)...")
batch = []
for chunk in chunks:
batch.append({
"id": chunk["id"],
"libro_id": chunk["libro_id"],
"text": chunk["text"],
"page_start": chunk["page_start"],
"page_end": chunk["page_end"],
"word_count": chunk["word_count"]
})
if len(batch) >= 100:
_upload_batch(batch)
total += len(batch)
print(f" {total} chunks subidos...")
batch = []
if batch:
_upload_batch(batch)
total += len(batch)
print(f" {libro['titulo']} completo")
print(f"\nTotal: {total} chunks subidos a Neo4j")
def _upload_batch(batch):
"""Sube un batch de chunks con UNWIND para eficiencia."""
run_write("""
UNWIND $batch AS chunk
MERGE (c:Chunk {id: chunk.id})
SET c.libro_id = chunk.libro_id,
c.text = chunk.text,
c.page_start = chunk.page_start,
c.page_end = chunk.page_end,
c.word_count = chunk.word_count
""", {"batch": batch})
if __name__ == "__main__":
upload_all()

273
vectorize.py Normal file
View File

@ -0,0 +1,273 @@
"""Vectoriza chunks y los sube a Neo4j con embeddings para vector search."""
import json
import os
import sys
import time
from db import run_write, run_query
from dotenv import load_dotenv
load_dotenv()
PARSED_DIR = os.path.join(os.path.dirname(__file__), "parsed")
CATALOG_PATH = os.path.join(os.path.dirname(__file__), "catalog.json")
# Embedding config
EMBEDDING_MODEL = "gemini-embedding-2-preview"
EMBEDDING_DIMS = 3072
GCP_API_KEY = os.getenv("GCP_API_KEY", "")
BATCH_SIZE = 10 # Smaller batches for gemini-embedding-2 (token limits)
def load_catalog():
with open(CATALOG_PATH, "r", encoding="utf-8") as f:
return json.load(f)
def save_catalog(catalog):
with open(CATALOG_PATH, "w", encoding="utf-8") as f:
json.dump(catalog, f, ensure_ascii=False, indent=2)
def init_embeddings():
"""Inicializa cliente de embeddings (google-genai con API key)."""
from google import genai
client = genai.Client(api_key=GCP_API_KEY)
print(f" Embedding model: {EMBEDDING_MODEL} ({EMBEDDING_DIMS} dims)")
return client
def create_vector_index():
"""Crea el indice vectorial en Neo4j si no existe."""
try:
run_write(f"""
CREATE VECTOR INDEX chunk_embeddings IF NOT EXISTS
FOR (c:Chunk)
ON (c.embedding)
OPTIONS {{
indexConfig: {{
`vector.dimensions`: {EMBEDDING_DIMS},
`vector.similarity_function`: 'cosine'
}}
}}
""")
print(f" Indice vectorial creado/verificado ({EMBEDDING_DIMS} dims)")
except Exception as e:
if "already exists" in str(e).lower() or "equivalent" in str(e).lower():
print(" Indice vectorial ya existe")
else:
print(f" Advertencia indice: {e}")
def build_embedding_text(chunk: dict) -> str:
"""Prepend metadata al texto para embedding contextual (v2).
Esto mejora la calidad del embedding porque reduce ambigüedad semántica.
Ej: "Capítulo: Nefrología. Sección: Inmunodepresores. <texto>"
"""
parts = []
if chunk.get("titulo_capitulo"):
parts.append(f"Capítulo: {chunk['titulo_capitulo']}")
if chunk.get("titulo_seccion"):
parts.append(f"Sección: {chunk['titulo_seccion']}")
if chunk.get("tipo_contenido") and chunk["tipo_contenido"] != "body":
parts.append(f"Tipo: {chunk['tipo_contenido']}")
prefix = ". ".join(parts)
return f"{prefix}. {chunk['text']}" if prefix else chunk["text"]
def generate_embeddings(client, texts: list) -> list:
"""Genera embeddings con gemini-embedding-2-preview via google-genai."""
truncated = [t[:2000] if len(t) > 2000 else t for t in texts]
result = client.models.embed_content(
model=EMBEDDING_MODEL,
contents=truncated,
)
return [e.values for e in result.embeddings]
def vectorize_libro(libro_id: str, model=None, use_v2: bool = None,
on_progress: callable = None) -> dict:
"""Vectoriza chunks de un libro que ya están en Neo4j (les agrega embedding).
Soporta tanto chunks v1 (text plano) como v2 (con metadata contextual).
Args:
libro_id: ID del libro
model: Modelo de embeddings (se inicializa si no se pasa)
use_v2: Si True, usa build_embedding_text. Auto-detecta si None.
on_progress: Callback(step, pct, msg)
Returns:
dict con estadísticas: {total, skipped, errors}
"""
def report(pct, msg):
if on_progress:
on_progress("vectorize", pct, msg)
print(f" [{pct}%] {msg}")
# Obtener chunks sin embedding de Neo4j
chunks = run_query("""
MATCH (c:Chunk {libro_id: $lid})
WHERE c.embedding IS NULL
RETURN c.id AS id, c.text AS text,
c.titulo_capitulo AS titulo_capitulo,
c.titulo_seccion AS titulo_seccion,
c.tipo_contenido AS tipo_contenido
ORDER BY c.chunk_index
""", {"lid": libro_id})
if not chunks:
report(100, f"{libro_id}: todos los chunks ya tienen embedding")
return {"total": 0, "skipped": 0, "errors": 0}
# Auto-detectar v2: si tiene titulo_capitulo, es v2
if use_v2 is None:
use_v2 = any(c.get("titulo_capitulo") for c in chunks)
report(0, f"{libro_id}: vectorizando {len(chunks)} chunks {'(v2 contextual)' if use_v2 else '(v1)'}")
if model is None:
model = init_embeddings()
total = 0
errors = 0
for i in range(0, len(chunks), BATCH_SIZE):
batch = chunks[i:i + BATCH_SIZE]
# Preparar textos para embedding
if use_v2:
texts = [build_embedding_text(c) for c in batch]
else:
texts = [c["text"] for c in batch]
# Generar embeddings
try:
embeddings = generate_embeddings(model, texts)
except Exception as e:
print(f" Error generando embeddings batch {i}: {e}")
time.sleep(5)
try:
embeddings = generate_embeddings(model, texts)
except Exception as e2:
print(f" Error retry: {e2}. Saltando batch.")
errors += len(batch)
continue
# Actualizar embedding en Neo4j (batch update)
updates = [{"id": c["id"], "embedding": emb}
for c, emb in zip(batch, embeddings)]
try:
run_write("""
UNWIND $updates AS u
MATCH (c:Chunk {id: u.id})
SET c.embedding = u.embedding
""", {"updates": updates})
total += len(batch)
except Exception as e:
print(f" Error subiendo batch {i}: {e}")
errors += len(batch)
pct = min(99, int((i + len(batch)) / len(chunks) * 100))
if (i + BATCH_SIZE) % 100 < BATCH_SIZE:
report(pct, f" {i + len(batch)}/{len(chunks)} embeddings")
# Rate limiting
if i + BATCH_SIZE < len(chunks):
time.sleep(1)
report(100, f"{libro_id}: {total} embeddings generados, {errors} errores")
return {"total": total, "skipped": 0, "errors": errors}
def vectorize_all():
"""Vectoriza todos los libros parseados."""
catalog = load_catalog()
print("Inicializando Google GenAI...")
model = init_embeddings()
print("Creando indice vectorial en Neo4j...")
create_vector_index()
total = 0
for libro in catalog["libros"]:
if libro["estado"] == "parseado":
print(f"\n{'='*60}")
print(f" {libro['titulo']} ({libro['chunks_generados']} chunks)")
print(f"{'='*60}")
n = vectorize_libro(libro["id"], model)
total += n
# Stats finales
stats = run_query("MATCH (c:Chunk) RETURN c.libro_id AS libro, count(c) AS chunks ORDER BY chunks DESC")
print(f"\n{'='*60}")
print(f" VECTORIZACION COMPLETA: {total} chunks nuevos")
print(f"{'='*60}")
for s in stats:
print(f" {s['libro']}: {s['chunks']} chunks en Neo4j")
def search_vector(query: str, top_k: int = 5, libro_id: str = None):
"""Busqueda semantica sobre los chunks vectorizados."""
model = init_embeddings()
query_embedding = generate_embeddings(model, [query])[0]
filter_clause = ""
params = {"embedding": query_embedding, "top_k": top_k}
if libro_id:
filter_clause = "WHERE c.libro_id = $libro_id"
params["libro_id"] = libro_id
results = run_query(f"""
CALL db.index.vector.queryNodes('chunk_embeddings', $top_k, $embedding)
YIELD node AS c, score
{filter_clause}
RETURN c.id AS id, c.libro_id AS libro, c.page_start AS pag_inicio,
c.page_end AS pag_fin, c.text AS texto, score
ORDER BY score DESC
LIMIT $top_k
""", params)
return results
if __name__ == "__main__":
args = sys.argv[1:]
if not args:
print("\nUso:")
print(" python vectorize.py all - Vectorizar todos los libros")
print(" python vectorize.py <libro_id> - Vectorizar un libro")
print(' python vectorize.py search "query" - Busqueda semantica')
print(' python vectorize.py search "query" --libro farreras-2020')
sys.exit(0)
if args[0] == "all":
vectorize_all()
elif args[0] == "search":
query = args[1] if len(args) > 1 else ""
libro_id = None
if "--libro" in args:
idx = args.index("--libro")
libro_id = args[idx + 1] if idx + 1 < len(args) else None
if not query:
print("Error: query vacia")
sys.exit(1)
print(f"\n Busqueda semantica: \"{query}\"")
results = search_vector(query, top_k=5, libro_id=libro_id)
if not results:
print(" Sin resultados")
else:
for i, r in enumerate(results):
print(f"\n [{i+1}] {r['libro']} - pags {r['pag_inicio']}-{r['pag_fin']} (score: {r['score']:.4f})")
preview = r['texto'][:300].replace('\n', ' ')
try:
print(f" {preview}...")
except UnicodeEncodeError:
print(f" {preview.encode('ascii', errors='replace').decode()}...")