Proxmox setting locale failed

If you log in to Proxmox’s shell and see error stating that “setting locale failed“, then execute the following command to get rid of that message:

dpkg-reconfigure locales

Not quite sure why it happens, but I see it almost all the time.

Recover pfSense from out of disk space and config.xml corruption

I use NetGate pfSense+ version 22 based on FreeBSD 21. I encountered a problem of running out of disk space because of packet logging enable in Suricata. Out of a sudden there was high traffic and therefore high logs production. Disk space utilization was over 100%. Now there is a problem with /config/config.xml file, there is no VPN, Suricata, pf configuration available from the UI. Dashboard is corrupted also. However there is /config/backup folder with backed up configuration files. I’ve taken the last known good with proper file size and put it in place. After reboot it works just fine. However I’m not quite sure if there are other issues out there.

To be sure not running out of space, increase disk space and reconfigure Suricata packet logging with proper file sizing to fit available space. If you are on bare metal, then chances are little to zero that you put too small drive in there. In case of virtualization, go to your configuration panel and increase drive space. Then go to serial console or SSH:

gpart show
gpart resize -i 2 da0
gpart show
service growfs onestart
df -h

After this you should have both partition and file system extended.

Ruby on Rails vs PostgreSQL 14 partitioning primary keys

Starting from PostgreSQL 10 we have available new type of paritioning, which is declarative one. Before we had inheritance which is also good but has some limitations. Now, with declarative partitioning user need not to specify partition name while inserting data. To create partitioned table:

CREATE TABLE tablename 
( 
  id BIGSERIAL NOT NULL, 
  created_at TIMESTAMP WITHOUT TIME ZONE NOT NULL,
  CONSTRAINT tablename_pkey PRIMARY KEY (id, created_at)
) PARTITION BY RANGE(created_at);

There are few things worth explaining. First one is BIGSERIAL data type. It is a bigger form of integer type with automatic sequence creation, which is very convenient. Second thing is PARTITION BY RANGE(columnname). It defined that the data will be spread across paritions based on created_at date range. It positively affects query performance.

To create a partition:

CREATE TABLE tabelename_2022_09 PARTITION OF tablename
    FOR VALUES FROM ('2022-09-01 00:00:00.00') TO ('2022-09-30 23:59:59.99');

With table inheritance I’ve used partitioning with over 1000 tables. Generally speaking, PostgreSQL, even older versions, could easily handle 100 000 tables on single server and data volume around 10 TB. However with declarative partitioning you need to remember, that when executing a query o master table there will be shared lock on each and every parition and it costs memory. Althought it migth be a little simplification you need to increase max_locks_per_transaction parameter to fit required number of partitions created, attached. If you do not need online access to some old data or created future paritions upfront then either do not attach them or just detach if unsed.

Last and the most important thing related to multi-column primary key. Ruby on Rails and ActiveRecord library does not support it by default. You could use some gem to overcome this limitation, but what you only need is to point the specific colum in model definition:

self.primary_key = :id

It works just fine.

Allow .local domain suffix to be resolved

On many chances you may need to configure some domains using .local suffix. Unfortunately this is covered by RFC 6762 (multicast DNS). On specific Linux distributions you will not be able to resolv such domains using DNS server you set either in the machine or in the cloud settings (for instance Azure).

How to you know that the DNS query is not reaching your desired DNS server? In case you use named, then first enable query logging by the following comman:

rndc querylog on

Now DNS queries from the clients who set that particular DNS server as its resolved will be visible in system logger file, like /var/log/messages. Then try to make a ping or curl to the desired domain address with .local suffix and check if it appears in the DNS server log. You can also force DNS query to reach that DNS server by using dig command:

dig yourdomain.local @yourdnsserver

If it’s present in DNS server log file then you’re confident that if you reconfigure your client machine it will work properly. To do this on Ubuntu 20 few things are required. First one is to have IP configuration fixed (some yaml file in /etc/netplan/ directory):

network:
    ethernets:
        eth0:
            dhcp4: no
            addresses:
            - 10.99.99.10/24
            gateway4: 10.99.99.1
            nameservers:
              addresses: [10.99.99.20]
            dhcp4-overrides:
                route-metric: 100
            dhcp6: false
            match:
                driver: hv_netvsc
                macaddress: 60:45:bd:94:4a:85
            set-name: eth0
    version: 2

And also as suggested in the netplan configuration file, to disable cloud init feature create file in /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following content:

network: {config: disabled}

Then reapply network configuration:

sudo netplan try

For me, personally, it is sometimes overcomplicated as comparing to CentOS distribution. Now with the fixed settings that should not be overwritten by some other processes you can proceed with disabling local DNS stub listener:

cd /etc/
sudo ln -sf ../run/systemd/resolve/resolv.conf
cd /etc/systemd/
sudo sed -i -e 's/#DNSStubListener=yes/DNSStubListener=no/' resolved.conf
sudo systemctl restart systemd-resolved

You can put your own desired DNS server in the /etc/systemd/resolved.conf file. After all of these operations try ping, curl or wget with your .local domain and check if this query appears in the DNS server log file.

Disabling uncategorized internal Suricata rules in pfSense

I want to enable only particular rules categories. Do not want to have all these internal Suricate rules as they cover too broad variaty of cases including loads of false-positive. If one would like to go for deep traffic analysis then they would be fine, but in case you see “STUN Binding Request On Non-Standard High Port” and know that is your P2P camera in the LAN then it’s worth disabling all of that things at once. To disable them it is a little bit tricky on pfSense installation.

Go to Interfaces and selected desired one. Be sure to uncheck all snort or ET rules before. Then go to WAN Rules and select Active Rules in category dropdown box. Click Disable All. Now you are running without all default rules and can enable only those which you are interested most to have. For example you can try with the following:

  • attach_response
  • botcc
  • 3coresec
  • ciarmy
  • compromised
  • deleted
  • dos
  • exploit / exploit_kit
  • hunting
  • malware
  • phishing
  • scan
  • shellcode
  • sql
  • threatview_CS_c2
  • tor
  • user_agent
  • web_client
  • web_server
  • web_specific_apps
  • worm

Multiple ZFS pools on single drive

Image a hypothetical scenario having two 512 GB drives and want to use a Proxmox ZFS VM replication onto a second server with one 1 TB drive. Solution is quite simple. By using fdisk, create two primary partitions on the bigger drive and then go to Disks.ZFS.CreateZFS and you will be able to select a partition for the particular pool. One downside of such a setup is that ZFS liks to have whole drive for it’s own, please keep in mind that the performance may vary.

disk/partition ‘/dev/sdX’ has a holder (500)

Having LVM on the disk causes system to automatically active such volumes. I installed used drive with Ubuntu on it to my Proxmox server. Using Proxmox’s UI you will not be able to wipe this drive, because volume group has been already auto-activated. You need to log into shell and then:

vgs # pick VG on your interest
vgchange -a n vgname # pass your VG to deactivate it

After this, you are ready to wipe drive from the UI.

Package is in a very bad inconsistent state

Aborting Ubuntu packages update via Ansible gave me some weird state of libatk-wrapper-java-jni package. My playbook installs default-jre and it was painfully slow so I thought that there is some lockup, but there was not. My VM performed poorly at that moment, probably due to the fact that it came from a HDD with badblocks which got migrated to another one. It just works, but I’m not 100% sure if it is healhty. So… in case you have some broken package installation try:

sudo apt-get install --reinstall libatk-wrapper-java-jni

And now you are good to go with autoclean, update and possibly upgrade to check whether really there is no problem still dangling on your system. On mine, it is fine now.

UnsupportedClassVersionError

Changing Java language level to run compiled code on older runtimes

I’m working on some Java project. I use IntelliJ IDEA and deciced to go with Oracle’s OpenJDK-18. But… this runtime is available by default only during compilation in the IDE. In the system I have OpenJDK-11. Trying to run code compiled by JDK-18 on JDK-11 gives me the following message:

java.lang.UnsupportedClassVersionError: Main has been compiled by a more recent version of the Java Runtime (class file version 62.0), this version of the Java Runtime only recognizes class file versions up to 55.

To overcome this go to module settings (or Project Structure) and change Language Level to lower value. In case you compile with 18 then it will have version 18 set. Switch to version 11 (local variables syntax for lambda parameters) and compile once again. Now you will be able to launch your application even with older JDK.