This new release of Meta-Press.es bring many speed improvements. First, the source definitions are now kept in the browser storage, this allows to avoid reading the 25 000 lines of the definitions at each new Meta-Press.es tab. Loading sources from the browser storage is 3x faster than reading them from my NVME drive (using Firefox Nightly ASAN, a slow version).

Then Meta-Press.es improved the way it performs the searches. The previous method was to query a source, wait for its answer and start over with the next source. The new method is to launch all queries at once (but no more than 500 per second to respect Quad9 recommandations) and only after that starting to study the responses. This allows to start waiting for the slowest sources right from the beginning, giving it more time to answer before actually delaying the request. On a french LTE connexion its 3x faster, with measured 620 sources fetched in 9s. This query scheduling improvement was achieve thanks to the microTask JavaScript API.

The last speed improvement is the multi-thread approach (thanks to the Wau Holland Foundation). You can turn it on from the settings, clicking the corresponding box. This allows to divide a search in batches, approximatively one by CPU core (minus a small percentage of spared cores), to get more CPU working on the search as JavaScript is otherwise single-threaded. This is done using different search tabs, automatically opened and closed. Results are gathered in the original tab. It might help in scenarios where the task is CPU-bound like on an old smartphones with good WiFi or an old computer with an optic fiber connexion. But usually, and with my LTE connexion for instance, the CPU is just waiting for the data to come from the network (so the option is not activated by default).

As per NLnet and NGI0 support, Meta-Press.es searches can now be run out of web browser, from the command-line, via a Deno script.

This is the final stage of the code architecture rework initiated on NLnet impulse to introduce a clean separation between the core functions and the interface. This way, most of the Meta-Press.es core code is reused in this new command line interface.

This should allow to embed Meta-Press.es in more situations, even server-side.

Currently, this implementation is not faster than the regular in-browser web-interface. It’s due to the fact that all web page analysis (aka as DOM operations) are emulated via the jsdom ^[1] library, while the DOM is native on web-browsers. Deno, nor NodeJS are giving access to a compiled DOM. So searches are CPU-bound when started from the command-line and run for minutes when single-threaded. Fortunately, the multi-threaded searches (done via the WebWorker API here ^[2]) are improving things a lot, getting searches on par with the web-interface on a big (16 threads) machine.

For instructions about the Meta-Press.es command-line interface, please check the corresponding project README section.