Un/check all items in iTunes 11

I subscribe to The Moth, and had previously selected all episodes to be sync'd to my iPhone. Recently I found myself short on space, so I wanted to tell iTunes 11 to stop doing that. Previously, it was a simple case of multiple selection in a list and uncheck, but with iTunes 11, it looked like I was going to have to do it all by hand... for over 200 episodes!

By playing around, I found that if you command-click on a checkbox, it has the effect of un/checking all the checkboxes in the list. This made things more manageable.



Unknown control sequence \doublespacing

If you are getting the error
"Unknown control sequence: \doublespacing" when you try to compile your 
beamer presentation in Lyx, change
Document→Settings→Text Layout→Line spacing 
to Single.



Notes On Running Calibre On A Server

Calibre is a great tool, especially for converting between ebook formats. Here are some notes for getting it to run on a headless server.
  • The binary installer off the website works fine — ignore warnings about completion and desktop integration failing.
  • The installer will always pollute /usr/bin regardless of the installation directly you choose.
  • You will need the following libraries for mobi conversion:
    • libxi6
    • libxrandr2
    • libxfixes3
    • libxcursor1
  • If you get this message
    • SVG rasterizer unavailable, SVG will not be converted
    • You will need to install xvfb and use xvfb-run, like so:
xvfb-run ebook-convert blah.epub blah.mobi


Nook vs Kindle

Definitely the Kindle. Amazon offers an amazing service, especially their personal document service's email feature. That is simply divine. Furthermore, Amazon isn't nearly so annoying to use as a non-US citizen. I don't have to use a trick credit card or proxies to purchase ebooks. Amazon just sells them to me, easy peasy.

Disclaimer: I have a first generation Nook and a Kindle Touch. My comments above are however entirely based on the services provided, not the devices themselves. So this is probably more appropriate as Barnes and Noble vs Amazon.



Thoughts on The Design of SPOT2 GPS

  1. I love the fact the screws holding down the battery cover has little flip up handles so you can turn them with just your fingers, and that they are set so that they don't fall out once loosened. This is top quality design. However, the battery compartment door is not attached to the unit, and it should be, so there is no risk of losing that. Admittedly, it is pretty hard to lose something that is fairly large and orange
  2. Somewhat more importantly, the SOS button should be hooked into the power button, in the sense that if you press and hold the SOS button, it turns the device on and goes into SOS mode. As it is, activating SOS is a 2 step process, requiring you to first turn the device on.



Jumpcut is dead. Long live Jumpcut

I used to use Jumpcut a long time ago, and was a big fan of it. At some point, I think around 10.5, it stopped working, and I let it go -- too busy at the time to poke at it.

Recently however my interest in it was piqued again, and downloaded the 0.63 source. To my great delight, it compiles just fine, and once recompiled, works perfectly. Score one for good programming and open source!

If like me you have given up on Jumpcut, it is time to add this nifty utility back to your toolbox.


P.S. If you don't have access to Xcode, or the idea of compiling a program sounds like a bit too much, drop me a line at freespace *at* gmail.com and I will make my copy available.


Acorn vs Pixelmator

Pixelmator wins hands down. Compared to Pixelmator Acorn is anaemic in terms of features and costs more than twice as much.

Disclosure:  I originally got Acorn as part of MacHeist, so I didn't pay full price for it. I did pay full price for Pixelmator.



Converting Excel 1.0 Files On OS X (or some other OS)

  1. Download SheepShaver
  2. Download "New World PPC ROM" from redundantrobot.com
  3. Extract the zip. This should produce newworld86.rom. Rename this file to ROM and put it in the same directory as SheepShaver.app
  4. At this point SheepShaver.app should run, showing you a folder with a blinking ? inside it
  5. Download "Mac OS 9 Boot Image (120 megs)", again from redundantrobot.com
  6. Unzip the downloaded file. This should produce OS9.img
  7. Access SheepShaver's Preferences, and add OS9.img to Volumes. Hit Save and restart SheepShaver
  8. SheepShaver should now boot into OS9. Ignore the warning about virtual memory
  9. On the desktop, there should be a Unix drive. This is your Mac's system volume. All your files can be accessed through this.
  10. Download Excel for Mac archive from Macintosh Garden
  11. Unzip, this should produce Microsoft Excel folder, inside you will find 5 versions of Excel
  12. Navigate to this folder OS9, and enter Excel 4.0 Folder
  13. Run Excel, and navigate to your Excel 1.0 files.
  14. Open each file, then save it again using Normal Format which is just Excel 3.0. Probably a good idea to add xls to avoid overwriting the original.
  15. Open the converted Excel 3.0 file in modern Excel. Microsoft Excel for Mac 2011 is capable of opening Excel 3.0 files.
At step 14 you might run into a problem I faced: Excel simply won't see any of the Excel 1.0 files! The problem is that Excel is using extended file attributes to determine which files it can open. On less sophisticated systems this is done via file extensions.

To see this, if you copy COMMANDS file out of Excel 1.03's sample directory (you have to convert the image first using Disk Utility), and run xattr on it, you will see this:

$ xattr COMMANDS 

Content of com.apple.FinderInfo is:

$ xattr -px com.apple.FinderInfo COMMANDS
58 4C 50 47 58 43 45 4C 01 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

This is the magic you need. Suppose you had a Excel 1.0 file called foo, you add this extended file attribute as follows:

$ xattr -wx com.apple.FinderInfo "$(xattr -px com.apple.FinderInfo COMMANDS)" foo

After doing this foo should show up in Excel inside ShavedSheep.

If all this sounds like too much for you, send me an email at freespace _at_ gmail.com with subject of "Excel 1.0 Conversion".


An alternative to using Sheep Shaver and OS 9 is to use Mini vMac and System 6. This will let you run Excel 1.0.
  1. Download Mini vMac
  2. Download "Mac Classic ROM" from redundantrobot.com. Extracting the zip produces vmac.rom which is incidentally the exact filename Mini vMac expects
  3. Follow these instructions to get System 6.0.8 running
  4. Drag-n-drop the Excel 1.0 program img file (see above) directly into Mini vMac. You will now have access to Excel 1.0
  5. To get a read/writable disk, create a dmg in Disk Utility. Drag-n-drop this into Mini vMac and you will be asked to initalize and format it. Do so, and then drag-n-drop Excel into this new read-writable disk.
  6. Do your conversion, shutdown Mini vMac
  7. Open the dmg, and it should mount, despite having been initalized and formatted by System 6.0.8 -- backward compatibility for the win
Mini vMac is not as powerful as Sheep Shaver, and System 6.0.8 is as much as it can handle. It is however much smaller and easier to use than Sheep Shaver, and for the purpose of converting Excel 1.0 to a more manageable format is more than sufficient.

Note that you might need to do the xattr trick above.



A Layman's Incomplete and Probably Inaccurate Understanding of The GFC

Foreword: It has been four years or so since the world went through the throes of financial upheaval known as the Great Financial Crisis. It was a complex event that will not doubt be analysed in depth by experts for years, if not centuries to come, as an example of what could happen. Sometimes we forget that we are still part of a grand experiment, and that nothing like our civilisation has ever existed before. This means that some lessons must be learnt the hard way. The recent LIBOR scandal however would suggests that the financiers are slow learners...

This is blog posts crystallises my thoughts and understanding of the event, gleaned from various places over the last few years. I make no claims on accuracy or veracity -- it is labelled rant after all.

Lastly, the terms I use below are only loosely connected to their actual meaning in the finance world. For example I use capital just so I can not use the word "money" so much. They mean the same thing here. And money just means something you can pay your debtors with.

Lets begin with some axioms:

  • There is a ceiling to the amount of money a bank can loan out
    • Therefore banks want to... realise the value of their loans as quickly as possible
    • They are therefore willing to sell some loans at a discount, in order to free up money which they can loan to someone else, so they can make more money
  • Perception of low interest rate is absolute, no relative. Thus if banks can borrow money at 1%, they can double the interest on their loans, i.e. 2%, and the perception of low interest rate is maintained. Whereas if the inter-bank loan rate was say, 20%, they can hardly add another percent without significantly affecting the number of people taking out loans.
    • If the profit banks make, derived from the difference between the interest rate at which they borrow money, and the interest rate at which they loan money, is high enough, then it is profitable to take on high risk loans. The increased risk of default is balanced by the greater reward.
Now for the play:
  • Easy credit meant more people are eligible to take out loans, most of which were used to buy houses
    • As a result, housing prices went up, and size of loans increased as a result
    • Some branches are opportunistic, and under-rated the risk of loan applicants so they can give them loans.
  • Banks started packaging high risk loans, which they were giving out due to the high profit margin, with low risk loans, and selling the entire package as low risk.
    • For example, suppose risk is 0..1, where 0 means the loan will definitely be paid back, and 1 means instant default. If one packages nine risk=0.1 loans with a single risk=0.5 loan, then the overall package was being sold with risk=0.14.
    • This practice inflates enormously the value of high risk loans, at the relatively small expense of low risk loans. 
  • These packages were then further manipulated. e.g. taken apart and with individual loans sold at the risk rating of the overall package. This serves to obscure the high risk nature of some of the loans in the package. 
    • For example, company A sells company B the above loans as a single package with risk=0.14. Company B then splits the package into five smaller portions, each with risk=0.14, and sells it to company C. If company C is lucky, it will be getting a good deal: two risk=0.1 loans instead of two risk=0.14 loans. If company C is unlucky, it will end up holding a risk=0.1 and a risk=0.5 loan.
    • These manipulation happened many times, to the point no one can be sure any longer of the true value of any individual loan.
Something happened at this point, I am not entirely clear what, but the effect was that financiers realised that some of the loans they hold are not worth what they are paid, so...
  • They raised interests rates to compensate for the suddenly higher risk of the loans they are holding, and;
  • They tried to sell off those loans who providence cannot be verified.
  • These had the effect of:
    • Increasing the number of defaults, and thus repossessions. As loan holders sold off houses in an attempt to recoup their costs, over-supply of houses depressed the price of houses, bursting the real estate bubble.
      • This meant that loan holders could not recoup their costs, and is thus operating at a loss
    • Devaluing the worth of loans with questionable providence, even if they are in fact low risk, and thus good loans.
      • This meant that loan holders were "stuck" with their loans
  • Both of these combined meant that:
    • Loans were worth a fraction of what they were worth before
    • Loans could not be converted into liquid capital
This freezing of capital, and the devaluing of assets lead to panic in the financial world:
  • Creditors called in their debts, especially from those debtors who specialise in mortages
  • But as mentioned above, the debtor's can't pay: the loans they have purchased with the money they had borrowed could not be converted back into capital.
    • As a result, debtors, often large financial institutions, go bankrupt or require injection of capital, i.e. "bail out"
  • Over all effect: 
    • The collapse of major financial institutions
    • Erosion of trust in the financial systems
    • Devaluation of real estate
    • Increased homelessness as homes are repossessed
    • Increased unemployment as institutions closed
    • Slow down in the flow of credit as creditors become more reluctant to give out loans -- once bitten; twice shy.
    • Extreme devaluation in the assets and worth of investors, from banks to companies 
These effects were not just limited to the U.S. The inter-connected nature of modern financial markets meant that banks and institutions all over the world lost billions if not trillions of dollars in worth, and the reduction in available credit pushed up the interests rates.

More importantly, this was not limited to private companies. Many banks are owned nationally, and thus governments around the world suffer in much the same way as private enterprises:
  • Large part of their worth disappeared
  • Higher interests rates
  • Creditors calling in their debts
Governments also have to contend with:
  • Lower tax revenue
  • Cost of bailouts to maintain social structure



Over this weekend I have been working on faerii, a USB controlled RGB LED based around the ATtiny85.

Here is a picture of the breadboard prototype

faerii breadboard prototype
The ATtiny85 is mid centre, and immediately above it is the ISP header. To the left of the ISP header, near the centre, is a 4 pin header that provides USB connectivity. Obscuring it is two filter capacitors, pull resistors, and zener diodes. Finally on the far left edge is a common-anode RGB LED.

Here is a video of it in operation:

The project is based around the hid-data example from vusb. The EEPROM contains control blocks, consisting of 4 bytes. The first 3 bytes specify the red, green and blue intensities, in that order. The last byte specifies a duration. Upon reading a control block, the LED transitions, over the specified duration, from its current colour to the new colour. Currently the duration specified in multiples of 20ms.

Lists of control blocks, which I call a control sequence, can be delimited using delimiter blocks. These blocks have a specified duration value of 0xff. When such a block is encountered, the sequence is restarted  from the beginning. Multiple sequences can be programmed into EEPROM.

The host software performs the job of programming sequences into EEPROM. It also provides additional commands, such as RESET and GOTO. The latter allows a collection of sequences to be stored in EEPROM, and activated individually. This feature was provided to minimise the amount of EEPROM writes required to change patterns. For example, sequences might be stored for an idle and new mail state. Activation then simply requires sending the correct GOTO command, without rewriting the EEPROM which as a limited write/erase cycle.

Currently a total of 62 control blocks can be stored. This is due to incomplete support for writing to arbitrary EEPROM address, and the fact the first block is reserved. It should be no great difficulty to fix this.

Two major features I have no implemented yet are REVERSE and MODIFY-ON-READ. These are animation options, where REVERSE specifies that upon reaching the end of a sequence, instead of starting again from the beginning, control blocks should be read backwards. MODIFY-ON-READ is more interesting. The intention is that after reading and executing a control block, the block's RGB  values should be modified. This allows free-running random colour cycling without explicit programming.

The firmware and commandline utility source code is hosted at github. I will add proper schematics next weekend, though you can probably figure most of it by googling and looking at the photo above.


GNU readline with System Python (OS X 10.7)

By default system python loads "fake" readline (fake because it actually uses libedit) from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so. Even if you install readline (which goes into /Library/Python/2.7/site-packages), it will still load it, even if you fiddle with sys.path.

The one solution I have found, which I am not terribly proud of, is to rename readline.so to readline.so.bak. This forces the GNU readline installed via pip/easy_install to be loaded.

Now I have nice things in the python interpreter again.



Notes on x86 Opcodes

Here are some notes I made while trying to understand the opcodes in this fun article on generating small elf binaries. Hopefully they will be of use.

  00000000 B801000000        mov        eax, 1
  00000005 BB2A000000        mov        ebx, 42
  0000000A CD80              int        0x80

In the above snippet:
  • B8 is the move instruction, in which the destination register is encoded into the opcode itself. The mov instruction is B8+r where r = 0..7, 0 being al/ax/eax depending on the data size. 
    • One might naively expect then that the next opcode to move 42 into ebx would be B9, and one would be wrong. The registers are not numbered alphabetically. The next 2 registers in sequence is cl/cx/ecx for r=1 and dl/dx/edx for r=2, and finally we get to bl/bx/ebx for r=3. This explains why the next mov instruction has value BB.
      • Note also the little endian encoding of bytes in the operand.

  00000000 31C0              xor        eax, eax
  00000002 40                inc        eax
  00000003 B32A              mov        bl, 42
  00000005 CD80              int        0x80

In the above snippet:
  • 31 encodes xor
    • C0 is the MOD-REG-R/M byte, which has the format:
      • 0..2: R/M (register or memory, 0..7)
      • 3..5: REG (register, 0..7)
      • 6..7: MOD (addressing mode, 0..4)
    • In this case, we have:
      • MOD=11
      • REG=0
      • R/M=0
      • Which says: addressing mode is register addressing mode, and the destination and source registers are eax (REG=0) and eax (R/M=0).
  • 40 is an instruction like B8, where the register to modify is encoded in the opcode itself. 
  • B3 is B0+r with r=3. This is like B8+r except B8+r deals with 16/32bit data while B0+r deals with 8bit data.
    • Recall again that the bl/bx/ebx is the 4th register, not the second.
Instruction information was sourced from the very useful X86 Opcode and Instruction Reference.



PSA: Using Lyx With Native OS X Spellchecker

Lyx 2.0 and onwards on OS X will use spellchecker provided by the OS X. Note however that the language it will use for spellchecking is determined by the document's language setting, not your system language setting.

Furthermore, in version 2.0.3, if you change your document's language setting, you have to close and re-open the document before it takes effect.



Notes on RF-0417C

Some notes on using the popular Bluetooth serial module RF-0417C:
  • When sending AT commands, do not include \r\n either together or singly.
  • It appears to be happy to receive 5V inputs to TX.
  • It is not happy to receive 5V to Vcc.
  • If you connect a LED, it blinks when unpaired, blinks slightly faster when paired, and becomes solid when a serial connection is established.
  • It works just fine with the SoftwareSerial library.



DFT and the Window Spectrum

In case I forget again why the window spectrum is computed when doing DFT, the answer is that discretisation  leads to aliasing artefacts.

Consider the DFT of a x(t)=1. Its DFT is calculated by calculating the sum sin(2 pi f t)*x(t) and cos(2 pi f t)*x(t) individually, then squaring the resulting sums, add them together, then  take the square root. Since x(t) = 1, for any frequency you are essentially summing cos(t) and sin(t), which you expect to be zero, and thus the power spectrum should be 0 for any frequency.

If you actually do this with, you will find that unless the points in time where sampling occurs covers an integer number of wavelengths, and is coherent in phase, i.e. sampling occurs in the same phase of each cycle, you will find that you don't get 0. You might get 0.1, 0.15, 0.09, 0.01, etc, but not 0. This is made more obvious by non-uniformely sampled data, such as those you might get in the real world.

This present a problem: the residue can produces peaks where there would otherwise be none, and this is more pronounced for lower frequency, because their wavelengths are longer and thus less likely to fit perfectly inside the window.

One way of rectifying this issue is to compute a correction coefficient c(f)=1-w(f) where f is the frequency, w(f) the window spectrum. The "corrected" power spectrum can then be calculated as correctp(f) = p(f) * c(f), where p(f) is the power spectrum.

I should add that the above method is not at all mathematical. It is chosen because any correction strategy that involves dividing by w(f) is unstable as sometimes w(f) gets very close to 0. Simply subtracting w(f) from p(f) won't work either because p(f)'s magnitude clearly depends on x(t) whereas  w(f) doesn't.

An obvious failure of the above correction is that it doesn't make the residue disappear - it only reduces it. A superior method would correct the DFT for a DC signal so it is 0 everywhere.


The obvious answer is to simply normalise x(t) so the maximum amplitude is 1.0. Then the window power can simply be subtracted. Duh.



PSA: Replacing the SuperDrive With A Hard Disk

Replacing the SuperDrive with a hard disk seems to be a Bad Idea at least in my 13" mid-2010 MBP: my 320GB WD Scorpio Blue took over a minute to copy 60MB. That is so 80s.

Interestingly, when I put the hard drive back to where it belongs, and put a SSD in the slot it vacated, the performance of the hard drive went back up to nominal.

Any one planning to upgrade to a dual SSD+HDD configuration should leave the HDD where it is and replace SuperDrive with the SSD.



Getting pyglet setup in OS X 10.7.2

  1. Create a virtual env and activate it
  2. hg clone https://code.google.com/p/pyglet/
  3. pushd pyglet; python setup.py install
  4. popd
  5. easy_install  pyobjc==2.3 # the ==2.3 is required as of 2012-01-27, or pyobjc-core will not install
  6. python -c "import pyglet" # this step should not fail