Architecture
Querido Diário deals with different challenges – described in Facing the data desert – and employs various resources to overcome each of them, in order to fulfill the Pillars of the project.
Its architecture is, therefore, a reflection of the technical decisions that connect these two ends: the set of solutions necessary to face the obstacles imposed by the availability of official gazettes and the interest in opening this data.
We can summarize the flow of the project for each type of official gazette in the steps below and understand them in more detail later.
Collect: obtain official gazette files from the source, the publishing websites
Process: apply treatments to the original files obtained
Provide: allow access and search on the stored contents
1. Data collection
Here, scraper robots are created to periodically collect official gazette files and associated metadata, such as publication date and edition number, from the publishing websites. Regardless of the type of official gazette, all of them require a scraper for file collection.
Production storage: DO Spaces for files and PostgreSQL for metadata.
Code repository: https://github.com/okfn-brasil/querido-diario
- Collection scheduling:
Historical series available on the website: once, when the municipality is integrated into the project.
Most recent edition: every day at 6 p.m., Brasília time.
Previous month (redundancy): every 1st day of the month at 8 p.m., Brasília time.
Learn more: Documentation on Scrapers
2. Data processing
This is the heart of Querido Diário, the stage responsible for text operations and for connecting the other components of the architecture.
Here, plain text is extracted from the official gazettes, saved in open TXT format along with the original file, and its content and metadata are indexed in the search engine to enable queries on the texts.
Note
Even though Text PDFs are the most common format provided by municipalities, this stage can also extract text from DOC and DOCX files.
This is also the stage where the integration of aggregated gazettes is implemented via segmenters. These are tools that slice the portions related to each municipality, from the various ones present in the file’s content, and produce individual TXT files for each municipality with the corresponding excerpts.
Finally, thematic filtering is performed, where searches are made in the primary index of the gazettes, to then create indexes with excerpts associated with specific themes like “Technologies in Education” and “Environmental Policies”.
Warning
Currently, the project does not integrate cases where the file format is Image PDF due to the low quality of text extraction from images (OCR). Similarly, fragmented gazettes are not integrated due to database modeling limitations.
Development: Python, Apache Tika for text extraction and OpenSearch for thematic filters.
Production storage: DO Spaces for files, PostgreSQL for metadata, and OpenSearch as the search engine.
Code repository: https://github.com/okfn-brasil/querido-diario-data-processing
3. Data providing
API
The Public API is the gateway through which access to the project’s data is facilitated. It has several endpoints through which different data can be requested from the official gazette databases and CNPJs from the federal taxes service (Receita Federal do Brasil, in Portuguese).
Development: Python using the FastAPI framework and automatic documentation with Swagger.
Production storage: PostgreSQL for company metadata and OpenSearch for gazette searches.
Code repository: https://github.com/okfn-brasil/querido-diario-api
Learn more: Public API Resources
Backend
The backend is responsible for being the intermediary mechanism between the Querido Diário frontend and its Public API when necessary. As users navigate the site, some of their actions trigger this mechanism to request data from the Public API, perform operations, and deliver the results to the site.
Development: Python, Django REST Framework and asynchronous task management with Celery.
Production storage: PostgreSQL for user and alert management and Redis for asynchronous tasks.
Code repository: https://github.com/okfn-brasil/querido-diario-backend
Web interface
This is the official Querido Diário’s website, its visual interface for users via Internet navigation. It contains the search interface for official gazettes. The site also serves as a dissemination tool for the project: it records its history and the people who are part of it, as well as producing knowledge around it, such as analyses, reports, and research.
Access: https://queridodiario.ok.org.br
Development: TypeScript and Angular framework
Code repository: https://github.com/okfn-brasil/querido-diario-frontend
Learn more: Search Interface Resources
Support
The official technical documentation of the project is meant to help the understanding of interested people, users, and developers, recording essential and more deeply described information on how Querido Diário works.
As it also serves as a “conversation between the project and its contributors,” the documentation repository contains the project’s roadmap (Roadmap).
Development: Sphinx
Code repository: https://github.com/okfn-brasil/comunidade
Learn more: Documentation on Documentation
Architecture design