May 24
2025
I have been working with industrial cameras and frame-grabbers that simply gives me a dump of their sensor data. To make that data useful it needs to undergo demosaicing, ideally using one of the modern demosaic algorithms like RCD rather than simple interpolation or the rather outdated VNG. Unfortunately the latter are all that OpenCV supports.
To gain access to more advanced demosaic algorithms and sophisticated image processing functions like colour calibration, micro-contrast, dehaze, etc, I encoded the raw colour filter array (CFA) data into DNG using the PiDNG library. With compression disabled this library produces DNGs which is accepted by darktable. However if I enable compression darktable is unable to open it.
Running darktable from the command line, the following error is observed:
71.5724 [rawspeed] (compression-test.dng) void rawspeed::AbstractDngDecompressor::decompress() const,
line 250: Too many errors encountered. Giving up. First Error: virtual Buffer::size_type
rawspeed::LJpegDecoder::decodeScan(), line 108: Unsupported predictor mode: 6
This suggests that the predictor used is not supported. By editing the source code of PiDNG, predictor was set to 1 which results in yet another error:
10.7300 [rawspeed] (compression-test.dng) void rawspeed::AbstractDngDecompressor::decompress() const,
line 250: Too many errors encountered. Giving up. First Error: virtual Buffer::size_type
rawspeed::LJpegDecoder::decodeScan(), line 141: Maximal output tile size is not a multiple of LJpeg frame size
This error happens because PiDNG uses an optimisation trick to improve the compression ratio. Consider a typical CFA pattern:
The raw sensor data is essentially the above repeated horizontally and vertically:
RGRGRG
GBGBGB
RGRGRG
GBGBGB
RGRGRG
GBGBGB
Lossless JPEG compressed by using a predictor that uses previous pixel values, including those in the previous row. Such a predictor work better if correlated values are closer together. So instead of saving the image as-is, the DNG specification (as of 1.4.0) allows the image data to be stored as an image that is twice as wide but halve as high (preserving the total number of pixels):
RGRGRGGBGBGB
RGRGRGGBGBGB
RGRGRGGBGBGB
In this new image, correlated pixels are closer together. e.g. the first red pixel had no immediate neighbours that was also red, but now it has an immediate neighbour in the next row. The rawspeed library however doesn't support this optimisation trick (or predictors other than 1), which to be fair is also useless with predictor 1, which only looks at the previous value on the same row. This trick is most useful with predictors that look at values from previous rows, e.g. mode 6 which is the default mode used by PiDNG.
With another modification to the PiDNG library that encodes the sensor data as a JPEG image of the original size, we were able to produce a losslessly compressed DNG file that darktable opens. The compression ratio isn't as good - 80% vs 75%. Hopefully darktable will gain support for predictors 2-7 and we can get better compression. In the meantime I cleaned up the changes to PiDNG and have submitted a PR which will hopefully be accepted.
Rant
DNG v1.4.0 was released in 2012, a time when DEFLATE compression in TIFF files was widely used yet for some reason the DNG specification restricts its use to floating point image data with estoic lossless JPEG, defined 20 years ago in 1992, being the only other lossless compression option. It is not until 2023, with DNG v1.7.0, that JPEG-XL is added for integer image data. Unfortunately JPEG-XL is exotic enough that darktable still doesn't support it in 2025. Why we couldn't have allowed Deflate for integer image data is beyond me. So much space could have been be saved.
Apr 10
2025
Because KeepassXC lacks the ability to decode QR code for TOTP setup and not all MFA implementations show the required secret in text format, I use the following snipet to read the QR code information locally:
python -c 'from cv2 import QRCodeDetector; from PIL.ImageGrab import grabclipboard; import numpy as …
Mar 28
2023
I love tinyproxy, it is simple to setup and more than enough for my needs. Recently however I noticed that some clients were experiencing unexpected long delays when doing something as simple as curl google.com. After some sleuthing, I tracked it down to the fact I had
Mar 28
2023
The default conda connect timeout is 9.15 seconds (no idea why this is, seems to have been the case from day one). If the proxy is slow, e.g. it is in another country or AWS region, this timeout can be triggered and conda will prematurely close the connection …
Feb 04
2023
In order to install the arcgis package in conda, you have to specify the architecture osx-64 otherwise the package will not be found. This works because MacOS will capable of running x86_64 code on Apple Silicon processors - that is how efficient they are!
conda install -c esri/osx-64 arcgis
In …
Jan 10
2023
Backups are a great idea, and encrypted backups are a even better idea. When backing up iOS devices on macOS, it is possible to set an encryption key to protect local files. There is however no straightforward GUI method to verify the encryption key.
Enter mvt:
Dec 20
2022
Went on a journey trying to figure out what valid exif tag types are, in both JPEG and TIFF files. In summary:
| TYPE # |
Name |
Exif 2.32 |
Tiff 6.0 |
Adobe TN[1] |
| 1 |
BYTE |
X
|
X |
|
| 2 |
ASCII |
X |
X |
|
| 3 |
SHORT |
X |
X |
|
| 4 |
LONG
|
X |
X |
|
| 5 … |
Dec 20
2022
Due to unexpected power loss or disconnection of the underlying storage, the filesystem inside Veracrypt volumes is desired to prevent future data loss.
On macOS I tried using
diskutil verifyDisk /dev/diskXXX
This didn't work for some reason, most likely because of some interaction with veracrypt. Same results via Disk …
Jul 30
2022
A quick systemd service definition file for tigervnc on Ubuntu 20.04
[Unit]
Description=tigervnc
[Service]
Type=forking
ExecStart=/usr/bin/vncserver -localhost no -depth 24 -geometry 1920x1080 -noreset
User=steve
[Install]
WantedBy=multi-user.target
Key components:
Type=forking so systemd correctly identifies the real VNC process, not the perl …
Jul 21
2022
Assembly4 and SubShapeBinders make it possible to assemble Parts, then using their position in the assembly to generate new Parts. However because new Parts depend on Parts in the assembly, if you try to add them to the assembly FreeCAD 0.19 will complain about cyclic redundancy, which makes sense …
Jul 21
2022
Scenario
- There exists a policy that snapshots directory ABC
- Directory ABC now no longer exists
- You want to keep existing snapshots
- You want to stop kopia from attempting to take any more snapshots
In order to achieve (4):
- Set the policy to manual only
- Disable inheritance from parent/global policy …
Jan 02
2022
I have been using SpiderOak One since Edward Snowden recommended it in 2013, almost a decade ago. I have over 1.5T of deduplicated data spread across 6 or so devices. This year, in 2022, I will not be renewing my subscriptions.
The main issues:
- Lack of updates: the last …
May 07
2016
I needed to read Lecroy binary waveforms files recently, and the one python script I had found didn't read them properly due to improper handling of 16 bit samples.
I wrote an alternative lecroy.py that should be a drop-in replacement. It handles 16 bit waveforms and exports a LecroyBinaryWaveform …
Apr 16
2016
Arduino-Makefile needed coaxing to in order to compile code targeting the SparkFun Pro Micro 3.3V. A major source of headache is the requirement for BOARD_SUB which isn't documented and required diving into the source code, and also the fact the Pro Micro has 2 PIDs for each variant (5V …
Apr 12
2016
A common pitfall of trying to use message forwarding in Objective-C is forgetting to implement
- (NSMethodSignature *)methodSignatureForSelector:(SEL)aSelector
in addition to
- (void)forwardInvocation:(NSInvocation *)anInvocation
Another problem is that sometimes calling objects will use
- (BOOL)respondsToSelector:(SEL)aSelector
to determine if your proxy object can respond to a particular …
Mar 23
2016
I recently started having trouble with Arduino 1.6.x not being able to program Arduino Micro or SparkFun Pro Micro, with avrdude complaining about permission denied when trying to open /dev/ACM0.
The usual solutions to this is to add oneself to the dialup group, or mess around in …
Mar 23
2016
Some issues I ran into when trying to move a Linux install from a 500 GB drive to a smaller 120 G SSD:
- When duplicating the filesystem, make sure that /proc exists, otherwise when you boot the new drive the kernel will complaint that /proc is missing and it can't …
Mar 23
2016
I recently acquired a used Lenovo X220 for use as a linux laptop, and needed the following in my openbox configuration XML to make the hardware speaker and mic mute buttons work:
<!-- Modified for X220 -->
<keybind key="XF86AudioMute">
<action name="Execute">
<command>pactl set-sink-mute 0 toggle</command>
</action>
</keybind>
<keybind …
Oct 04
2015
Introduction
I recently started playing around with software defined radio using a USB TV tuner dongle utilising the popular RTL2832U chipset. After playing around with software like
CubicSDR and
gqrx I was somewhat frustrated at the opaqueness of what is going on under the hood. As such I resolved to …
May 03
2015
I bought a Seeedstudio Xadow recently, and to get it to work with Arduino 1.6.3 was bit of a pain. Long story short, you can find the necessary files along with pithy instructions at my github repo.
The most crucial difference between the files Seeedstudio supplied and what …