Decentralized search engine & automatized press reviews

Version 1.8.0 : new source list, user feedback

1. Improving code architecture

After a long work here is the version 1.8.0. It required longer than usual as I integrated work from interns. It lead me to establish coding style rules. It’s also the reflect of a big "under the surface" work to migrate the code base to the modern modular design of JavaScript. is now following this architecture, which helped to minimize redundancies in the code (you want redundancies in your hardware, but note in your code). This will help for the maintenance of the application.

2. Managing relative dates for all sources

Another "invisible" change occurred regarding the possible date format definitions in sources. Now documented here, this evolution of the source definitions allows to handle as many formats as needed for each source. This is the solution of the different relative date format that we could find on sources (such as : Published one hour ago) which no JavaScript date libraries were able to handle in different languages. This evolution is not a generic solution for everyone like month_nb but it allows to handle several problems with dates (such as relative dates or inconsistencies betweens dates in archives of a same source…).

3. Major improvements of the release

This new version also brings visible improvements:

  • export / import in CSV format (to easily reuse results)

  • per-source feedback mechanism (to report broken sources)

  • new list of sources (to fine tune source selection)

3.1. CSV format

CSV format appeared in the import and export drop-down format menus. It currently only includes results (excluding research parameters that can be exported via RSS, ATOM or JSON).

3.2. User feedback about sources

The source feedback mechanism is a new button that appears when hovering a source in the searched source list of a finished research (also knowns as meta finding list, first box at the right of finished search). Clicking on this button open a pop-in frame that allows to describe the problem encountered with this source and to send your feedback in one click. This is the only way to send a request to the server from the application.

From a technical point of view, this generates 404 HTTPS requests that I’ll review using goaccess, my web log analyser tool. This way, everything gets automatically erased (with the rest of the logs) after the default Debian erasing log period of time (currently 15 days). This way, with a minimal effort it’s goaccess that will perform on demand statistics over the feedbacks, allowing to focus on the most reported sources. Information must flow.

3.3. New source list and cherry-pick source selections

The new source list answers two needs:

  • users did ask for a human readable list of the available sources

  • this list is the new source cherry-pick mechanism

From this new source list you’ll be able to edit the tag-based source-selection. For instance, you chose to search only in french sources (via the french language tag), you can then click on the new list sources button (situated under the two rows of the tag selection mechanism) and you’ll open a new frame, with 4 tabs.

  • The 1st tab lists all the sources

  • the second tab lists only the selected sources

  • the 3rd tab lists only the explicitly added sources to the selection

  • the 4th tab lists only the explicitly removed sources from the selection

So you can add some English sources to your french selection, or remove some french sources.

On a technical point of view, sources are selected from their tags, and a remove sources from the selection is applied, then an add source to selection list. You can edit those lists and they are exported with RSS, ATOM or JSON exports, they are added to the parameters of the permalinks and so saved for your scheduled searches.

In each tabs you can filter out the content via a search input, or act on batches (all the listed sources or just the current page) to add or remove sources to / from the current selection.

So you can visually check in which sources the next search will perform and then you can edit this selection.

Those parameters will be kept for your next searches and the source list will be automatically re-opened if you are using an edited source selection to search in, at least to help you remembering the choices.

Also for each source in this list, a magnifier button allows you to search only in this source, in one click.

4. Other improvements (ergonomics, source fixes…)

This new release also brings presentation improvements : results are presented with more density, their background color now alternates to help the reading and a lot of small glitches have been fixed (button sizes, margins…).

The source creation page have been improved to allow searching in the source definitions listed here (the row JSON file of all the provided source definition accessible here, and searchable from Firefox CTRL+F mechanism).

The previous released version introduced a source-definition automatic reloading mechanism when tabs where let down for others (such as Custom Source ones) and so your custom sources where automatically reloaded in a quite slow way. The naive implementation have been improved to reload sources only if custom sources where actually modified.

This previous version also had a bug regarding all the RSS based sources (±25%) which is now fixed along with a lot of source fixes. So we are, at date, with more than 315 working sources.

5. What will come next ?

There is still of work to do, to improve source testing for instance, or to automate releases testing.

But I also have some new features in mind, such as:

  • a child mode to lock the source selection on child compatible ones to help schools using

  • a dynamic form to allow regular users to create new sources using their mouse only (in 80% of the cases)

  • a port of as a regular Android application (to target more audience)

6. Donate to help the project

To finish, every donations would be warmly welcome to push the project further, or to thanks for this great tip : Access content of 7400 from the French national library for 15 € / year.