Architecture

Querido Diário deals with different challenges – described in Facing the data desert – and employs various resources to overcome each of them, in order to fulfill the Pillars of the project.

Its architecture is, therefore, a reflection of the technical decisions that connect these two ends: the set of solutions necessary to face the obstacles imposed by the availability of official gazettes and the interest in opening this data.

We can summarize the flow of the project for each type of official gazette in the steps below and understand them in more detail later.

  1. Collect: obtain official gazette files from the source, the publishing websites

  2. Process: apply treatments to the original files obtained

  3. Provide: allow access and search on the stored contents

[TODO]

1. Data collection

Here, scraper robots are created to periodically collect official gazette files and associated metadata, such as publication date and edition number, from the publishing websites. Regardless of the type of official gazette, all of them require a scraper for file collection.

  • Development: Python using the Scrapy framework.

  • Production storage: DO Spaces for files and PostgreSQL for metadata.

  • Code repository: https://github.com/okfn-brasil/querido-diario

  • Collection scheduling:
    • Historical series available on the website: once, when the municipality is integrated into the project.

    • Most recent edition: every day at 6 p.m., Brasília time.

    • Previous month (redundancy): every 1st day of the month at 8 p.m., Brasília time.

  • Learn more: Documentation on Scrapers

2. Data processing

This is the heart of Querido Diário, the stage responsible for text operations and for connecting the other components of the architecture.

Here, plain text is extracted from the official gazettes, saved in open TXT format along with the original file, and its content and metadata are indexed in the search engine to enable queries on the texts.

Note

Even though Text PDFs are the most common format provided by municipalities, this stage can also extract text from DOC and DOCX files.

This is also the stage where the integration of aggregated gazettes is implemented via segmenters. These are tools that slice the portions related to each municipality, from the various ones present in the file’s content, and produce individual TXT files for each municipality with the corresponding excerpts.

Finally, thematic filtering is performed, where searches are made in the primary index of the gazettes, to then create indexes with excerpts associated with specific themes like “Technologies in Education” and “Environmental Policies”.

Warning

Currently, the project does not integrate cases where the file format is Image PDF due to the low quality of text extraction from images (OCR). Similarly, fragmented gazettes are not integrated due to database modeling limitations.

3. Data providing

API

The Public API is the gateway through which access to the project’s data is facilitated. It has several endpoints through which different data can be requested from the official gazette databases and CNPJs from the federal taxes service (Receita Federal do Brasil, in Portuguese).

Backend

The backend is responsible for being the intermediary mechanism between the Querido Diário frontend and its Public API when necessary. As users navigate the site, some of their actions trigger this mechanism to request data from the Public API, perform operations, and deliver the results to the site.

Web interface

This is the official Querido Diário’s website, its visual interface for users via Internet navigation. It contains the search interface for official gazettes. The site also serves as a dissemination tool for the project: it records its history and the people who are part of it, as well as producing knowledge around it, such as analyses, reports, and research.

Support

The official technical documentation of the project is meant to help the understanding of interested people, users, and developers, recording essential and more deeply described information on how Querido Diário works.

As it also serves as a “conversation between the project and its contributors,” the documentation repository contains the project’s roadmap (Roadmap).

Architecture design

[TODO]