Year: 2023

Load OSM map files into PostgreSQL

You can use OpenStreetMaps on your own hardware. You need to grab map files first, which can be found at https://download.geofabrik.de. Once you downloaded it, install PostgreSQL and enable few extensions:

CREATE DATABASE yourdbname;
CREATE EXTENSION postgis;
CREATE EXTENSION hstore;

It may be useful at some point later on especially if you would like to try build your own tile server. Next you need to install osm2pgsql and it can be found in system packages in Ubuntu 22. Then:

osm2pgsql -c -d yourdbname -U youruser -H yourdbhost somefile.osm.pbf -W

Now give it database password and it will start loading. Depening on hardware it might take from few minutes to tens of hours in case of large files like Europe area. You should have something like this:

geography_columns
geometry_columns
planet_osm_line
planet_osm_point
planet_osm_polygon
planet_osm_roads
spatial_ref_sys

These are tables with points, lines, polygons and roads. Most of the data is located in the PBF file itself, but few other are calculated on the fly while loading. There is plenty of options for such data. You can use them in some desktop software like QGIS (loading directly PBF file) or setting up complete tile server similar as on the official OSM website.

Install Redash

Data is valuable if consumed or at least identified. For private and corporate usages I suggest installing Redash as it gives options for saving queries, exporting data, creating visualizations and dashboards and also setting up alerts. There are few other interesting features like creating dropdowns and inputs from saved queries or joining resuls from different data sources thru in-memory SQLite instance.

To install Redash, clone the repository. I recommend running Ubuntu 18 LTS server version as it is tested on this distribution. Then chmod a setup.sh file for execution and run it. It will ask for sudo password and going to install all the required things like packges and containers. It is based on Docker so in case of production setup better tweak it a little bit for volumes at least.

That’s all, your Redash should be instantly available to use.

WebGL performance

As already stated in my year 2023 plans, there will be some time for graphics programming. Back in the days I was exploring OpenGL but that was way long ago. Nowadays I think to try something much simpler and accessible like for instance WebGL. This library relies on JavaScript for logic and OpenGL ES for presentation layer. So it is hardware accelerated.

But there are some issues with this acceleration. On some computers you can specifically select particular GPU in BIOS/UEFI. However not on every computer. To start with I tried on Lenovo ThinkPad T420s with NVIDIA NVS 4200M. Testing on aquarium demo (which can be found here) I get stable 37 FPS on 5k elements with somewhere around 95 – 99% of load. There is sometimes some performance degradation when FPS count goes as low as 10. GPU and CPU temperature show the same value of 95 degress C. That is way too much. This computer has great advantage over others because it has NVIDIA Optimus feature which gives be ability to switch between graphics chips or let harware to choose itself which GPU should be used at the moment.

My other machine, which is Dell G15 lack such a feature and Ubuntu 22 picks integrated Intel UHD 10th gen GPU instead of heavy NVIDIA GeForce RTX 3050 Ti. So on this integrated chip we have stable 45 FPS, but without any performance drops. I am able to run Firefox browser on “discrete graphics” because I have this proprietary driver but it gives me only 50 FPS, so only 5 more than integrated one. It shows only 30% load on nvidia-smi so it is not doing that much. I tell you even more, on 30k elements there is more FPS on integrated GPU (20 vs 15). I do not quite get it what is going on here.

I still test it on various other harware configurations and it gives more weird unexpecting results.

Plans for 2023

I’m looking forward for new year’s technology opportunities. I have few ongoing projects which I would like to finish by the end of this year. Here is some brief overview of them:

  • Data Mining chapter of Simple HPC series
  • News feed tool as a subproject for data mining
  • AI/ML project utilizing news feed, OpenCL processing and user-input training
  • Highly portable system monitoring tool for my day-shift
  • Video graphics… most probably WebGL/THREE.js

I will try to fit all of these within just around 300 hours available…