We have an embedded system, which in its final form factor has only its power-cable, and networking port exposed. There are usb and hdmi ports in the machine, but these are concealed. The only access to this device is via networking(ssh). The SSD on this disk is factory imaged. During developement cycles, if we need to reimage and test, we need to open up the case to get access to the usb/hdmi ports for connecting monitor/keyboard and get console access — so that we can choose different boot options from BIOS and follow console instructions during boot. Typically this is done by booting a live-linux-image over USB and then from the console-terminal, we scpthe disk-image from a remote-file-server to the sda of the machine, and reboot the machine.

This article shows how to reimage this machine without using console/keyboard access — purely via networking access. This saves the trouble of having to open the casing and do the reimaging on the fly.

Challenges

The following things make the installation challenging:

• The choice of boot option is always set to the SSD disk in the UEFI. Editing this to anything else is challenging in the first place without access to UEFI console.
• Even if we manage to PXE boot instructing the UEFI to do so (say by tools like efibootmgr), we still dont have console access to follow through instructions as part of boot — we need to boot some live OS, and then ssh into the machine to dd the image. To know the IP of the machine running the liveOS, and the choice of user-name / passwd is not trivial.
• Further PXE boots call for extra options doled out as part of DHCP-responses — and this needs talking to the IT-personal as we dont own the corporate DHCP-server settings. Some non-intrusive way to doling out the boot images/settings, without having to adjust corporte-DHCP settings would be preferred.
• Using a non-PXE boot option, like trying to dd an image to the SSD, from an OS that is still mounted on the SSD itself, is not guaranteed to work. This is because random processes(eg systemd/journaling) of the running OS may continue to write to the disk and the SSD may have a corrupted image. So we are pretty much limited to performing the imaging of the disk when the disk is unmounted and is totally untouched by the running OS, except for the dd process.

Quick rundown of the solution

• We run a http-server on the network. This will serve iPXE boot-scripts, initrd/vmlinuz images, live-ubuntu-iso, and autoinstall scripts(that will just stop post kernel-load without really installing anything).
• Use iPXE. We will use a customized image that will simply chain load from a script off the http-server.
• On the current running OS, we will copy the customized iPXE image and let grub boot from this next time.
  – The UEFI on the device continues to boot from SSD. But the grub will now boot the iPXE file, instead of the current OS.
• The iPXE image will pick the script-file from the http-server, that further instructs it to boot the live-ubuntu-iso. Here we supply among other args two important args.
  – The first is the choce of IP. Once the live-ubuntu is booted, we pass a predictable IP so that we can ssh into the machine.
  – The second is the autoinstall script to use — this script contains a preset user-name and ssh-keyfile/password to use so that we can login to the live-linux that is just booted.
• Once the machine boots the live-os, we keep trying to ssh login (with the known credential). Once the ssh is successful, we dd the image from the file-server to the hard-disk.

The steps in detail

Preparing the iPXE image

Clone the iPXE repository

git clone git@github.com:ipxe/ipxe.git
cd ipxe/src

# prepare a file boot.ipxe that will simply pick a file from an external server
# ensure the change the IP of the server here. If the IP of the server changes
# we will have to rebuild this ipxe image
cat > boot.ipxe <<EOF
#!ipxe
dhcp
chain http://192.168.122.110:8999/boot-http.ipxe
EOF

make bin/ipxe.lkrn bin-x86_64-efi/ipxe.efi EMBED=boot.ipxe

As part of the make we will get a bin-x86_64-efi/ipxe.efi file.

Preparing the grub entry

On the current running machine that needs to be reimaged, add this content to /etc/grub.d/40_custom at its end. The choice of the UUID should preferably be the partition id where /boot is mounted. Copy the ipxe.efi file obtained in the above step in the root folder of this partition. Care should be take that the UUID reflects the partition where the ipxe.efi is kept and the chainloader argument reflects the path of the file in this partition.

menuentry "Reinstall via iPXE" {
  insmod part_gpt
  insmod chain
  search --no-floppy --fs-uuid --set=root 84d5ce35-af85-44fe-bd15-c26b8418d0d6
  chainloader /ipxe.efi
}

Once added, rebuild grub-cfg: sudo update-grub and find the position of this menuentry:

grep -e '^menuentry' -e '^submenu' /boot/grub/grub.cfg

Giving a 0-based index, say if the Reinstall via iPXEappears 4th in the list, do: sudo grub-reboot 3 to have grub choose this entry for next boot.

Preparing the HTTP server

On the machine that we chose to run http server, we will have to first pick a folder where we have all of these files

• Ubuntu-ISO file. The ubuntu-22.04.4-live-server-amd64.iso(jammy-server) would do fine.
• Pick the casper/initrd and casper/vmlinuz files out of iso and copy in the same folder. Use tools like ( 7z x file.iso ) to unzip the iso and extract the 2 files.
• Create a autoinstall file that supplies the username/password to login in via ssh. A sample file is shown below. We can keep all of the below same, except for the ssh_authorized_keys and the password under list. Use any password and ssh-key of choice. The password is a encrypted one. Use python3 -c ‘import crypt;print(crypt.crypt(“clearpassword"))’ to get this.
• The name of the file having the below content MUST be cloud-data. On the same place also touch meta-data and touch vendor-data. The latter 2 are empty files, but MUST exist!

#cloud-config
chpasswd:
    expire: false
    list:
        - installer:$6$TrC8TZduman3BRLs$KZd58FKOVZ8Hqc/ielAw.B/AcB20Ls.J3TD7IKiN53.i.iB0xtIyFN4CxKUnTTQwxcTPlA1kQgwrUUeeVF84w.
ssh_authorized_keys:
    - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg/6dxFHFbrQct3sdo42qG29Q5CAsUESwIUq6ZH7jTX/h9ayikqnDSNnKka1PGRcbAkMCQFvsFvEwjZiI7n76lhcsuyRfwRDQmpwMUnLIJNobGWTLscuhBXe7eg9WAR8cuN5hOE1b9UPmy2VYMJ0R2BydFMhSwQ9f4ZMLnqeooMHfj+x5k14jpCO0UZCmR82/esZO4QfeSzQWhaIbTQMZsNTRvn8qRObUo80bakUI4IK8oWZgbwqj2Z7QyA6mzqo4rTJpFyltho3VRiv9DBsPtYBh6YE0cguKkcB9Bzp9zrDDEfQIXnp8XS/W4tn1kS1478DPlABzPk+BA7ogaSxPVdGqy0Q5IAfzWuNNI6z5KFwYVuteqqR1hNF4dQGdq2B/aFIXygt0tSALkEoHnezFfE9WeK2wetxZAjtxqjsvLl8474qw0UjYWtVuQdavwUyjtS7EBZ44826ayLQGACHt54eNNTp45ReSC/AMeZ1znavSyMZceQCsWG5qDSd+vnNs= SSH-key-for-login"
system_info:
  default_user:
    name: installer
    lock_passwd: false
    gecos: Ubuntu
    groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash
autoinstall:
  version: 1
  early-commands:
    - ip addr show > /dev/ttyS0
    - ip addr show > /dev/tty0
    - while [ ! -f /run/some-non-exist-file ]; do sleep 1; done

• Create a boot-http.ipxe file that instructs to boot the live-server.
  - The ipval gives the IP-details of the machine that will be used when the live-os is booted. This is how we know which IP to login. We can choose to give the same current IP used in the system to avoid DHCP conflicts with other machines on the network. Its format is client-ip::gwip:netmask::::dnsip

set server_ip http://192.168.122.110:8999
kernel ${server_ip}/vmlinuz || sleep 3600
initrd ${server_ip}/initrd || sleep 3600
set ipval 192.168.122.100::192.168.122.1:255.255.255.0::::192.168.122.1
imgargs vmlinuz initrd=initrd boot=casper ip=${ipval} cloud-config-url=/dev/null url=${server_ip}/ubuntu-22.04.4-live-server-amd64.iso autoinstall ds=nocloud-net;s=${server_ip}/ --- || sleep 3600
boot || sleep 3600

• The || sleep 3600 is a way to see what failed. Otherwise iPXE will yield control back to UEFI and we can’t debug much. This assume we have console access — not a bad idea to have until the process is known to work smoothly without console.

A simple http-server can be started by running the following command on the folder having all files.

python -m http.server 8999

Reboot the machine

Now we can reboot the machine that needs to be reimaged. Once done, by way of grub setting, it will load the ipxe image. This will then auto-load the embedded boot.ipxe. The machine will get its ip over dhcp (some random ip). Then it will get boot-http.ipxe from the http-server. Then it will get the vmlinuz/initrd files and switch into that kernel. At this point, by way of kernel args, it will assign itself the IP that we had given and start booting the jammy live OS. Then by way of kernel-args it will start auto-installing ubuntu post the kernel boot-up. The autoinstall yaml contents are downloaded from the http-server. The auto-install file instructs the creation of a installer user with the choice of password/ssh-key and the early-commands simply holds the installation forever at the while..sleep loop. At this point, the machine is ready to be SSH’ed into. We can ssh into and then do whatever we want — which is to ddthe actual target image from a remote-machine.

ssh remote-file-server "cat /actualimage.gz" | sudo gunzip -c | sudo dd of=/dev/sda bs=1M status=progress

Once done, we reboot the machine, and we now reboot off the brand new image w/o using usb-sticks/console access!

Mac offers a cool pbcopy and pbaste command-line programs to exchange contents from/to clipboard right from command-line. This chains nicely with other unix tools using the standard stdin/stdout interface. For instance:

pbpaste > file.txt
ls -1 | pbcopy

Pasting MAC’s clipboard into the server is trivial as doing a ⌘v in iTerm will paste the current clipboard’s contents where we are in, be it a terminal on the local mac or the remote machine if we are currently ssh’ed there. However, loading something from the server into the MAC’s clipboard isn’t easy without reaching for the mouse. Further we should have the content displayed on screen before we can select it in iTerm and slurp into the clipboard with a ⌘c. This is frustrating for huge multi-page content. Extending command line copying from the server relieves us from using the mouse, and removes the artificial limitation of having to keep the content on screen. I find it very handy when we need to say get some code-snippet or output of some command from the server and paste it in mac’s slack app, outlook or browser.

The trick is simple. Just ssh into the MAC from the linux-server and execute pbcopy, giving the contents to ssh as its stdin. In essence, do

echo Whatever-needs-to-be-copied | ssh mac_user_name@MAC_IP pbcopy

A few steps are in order to do this seamlessly, though.

1. Setup the ssh-key-based login. Generate a ssh-rsa key-pair in the linux-server, and add the public key to the mac’s MAC-HOME/.ssh/authorized_keys. This way ssh gets non-interactive.
2. The main problem with ssh’ing back into mac from the linux-server is that the mac’s IP is ephemeral in nature. The MAC gets a different IP depending on its current locations — a wi-fi network, a wired-network, or in my home over VPN. This can be further complicated if the MAC is behind a firewall/NAT. The bottom line is that there is no well-known IP for the linux-server to SSH-into to reach MAC. To have a stable way of ssh’ing into the mac from the linux-server, no matter where the mac is, we use ssh’s tunneling option.

Run this ssh command from MAC

ssh -nNT -R 9222:localhost:22 my_linux_id@my_linux_server.myoffice.com &

The above command opens up a listener at localhost:9222 on the linux-server and forwards the packets to the MAC’s localhost:22. The -nNT options are useful for this tunnel-sake-only ssh. So, doing a

ssh -p 9222 mac_user_name@localhost

in the linux-server, ssh’s back into MAC. Voila! Needless to say, this will stop working once the ssh connection setup by the tunneling command is severed (happens when MAC moves out from a network, or if the MAC goes into standby..). We ust have to re-issue that command, usually when we find the clip-board isn’t working!

3. Have a bash function in the linux-server to enable copy-paste

to_mac_clip () {
   toSet=$(cat) 
   l=$(echo "$toSet" | wc -l) 
   echo -n "$toSet" | ssh -p 9222 mac_user_name@localhost pbcopy
   if [[ $l -le 3 ]]
   then
      echo "Your clip is setup with: $toSet"
   else
      echo "Your clip is setup with $l lines"
   fi
}

That’s it. Anytime, to copy to MAC form the linux-server, its now

echo any-command-generating-output-to-stdout | to_mac_clip

The same can be extended to work in vim too. Here is a vim-function and a key-map I use

function! DumpToClipBoard()
  if s:uname == "Darwin"
    call system("pbcopy -pboard general", getreg("\""))
  else
    call system("ssh -p 9222 mac_user_name@localhost pbcopy", getreg("\""))
  endif
endfunction

vmap <C-c> y:call DumpToClipBoard()<CR>

I usually visually select what I want and then do y. I now issue ctrl-cto copy to MAC’s clipboard, when needed. The above makes Vim first yank normally and then also push the contents of the default-register into the ssh-command, thereby bringing it to MAC’s clipboard.

I usually listen from Google Play. For this, there is an awesome desktop-google-play music app — that supports the media-play/pause key — so this is straight off the box (but having to move my hand off from home row to media key is still a sore).

Of late, I have started listening from Pandora.com. This means, its a chrome tab that is streaming music. Chrome unfortunately doesn’t respond to the media-key by itself. There is a chrome extension, (probably others similar too) — stream-keys, that helps to capture the media-play/pause key to stop/start music running from popular sites. I was using this for some time. However, it was becoming unresponsive frequently. Further, I wasn’t too happy with running yet another extension in the background in chrome. Manually switching to chrome, scan for the pandora tab, click it and pause/play the music wasn’t exactly fun. It is absolutely distracting and flow-breaking to to switch to a browser, especially a music streaming tab when we are deep into watching code/running tests in iTerm!

I was looking for alternatives — and came across this awesome gist, about using applescript to play/pause youtube. The script searches through all windows of chrome / all tabs of chrome and matches for the tab with the title - YouTube, grabs the Play/Pause button element using java-script and executes a click event on that element! Cool. This gist also suggests how to use Alfred to execute the script over a global-hotkey. I just replaced - YouTube with Pandora Radio for my use. Since I use quick-silver, instead of Alfred, I just used quick-sliver to fire up this script on a global-hotkey (and one of my choice — not the distant a-mile-away media keys!). This works great so far.

I liked one another feature of Google-Play Desktop app — ability to know the current song that is playing, from my terminal. The google-play desktop app, exposes the currently playing track’s information in a playback.json file, and we just need to look there and print the content from there. This way, when I like a song that I am listening and I want to know its title/artist, I dont get distracted off my terminal window by looking into the web-page/album-arts of the current song, but just dump that in terminal. I wished a similar interface was possible for the Pandora-Radio in my chrome-tab as well.

Reading here and there with my meagre applescript and javascript knowledge, finally pulled off this:

#!/bin/bash

result=$(osascript <<EOF
    tell application "Google Chrome"
        set songtitle to ""
        set songartist to ""
        set found_video to false
        set window_list to every window
        repeat with the_window in window_list
            if found_video is equal to true then
                exit repeat
            end if
            set tab_list to every tab in the_window
            repeat with the_tab in tab_list
                if the title of the_tab contains "Pandora Radio" then
                    tell the_tab
                    set songtitle to (execute javascript "var outputtitle = document.querySelector('[data-qa=\"mini_track_title\"]').innerHTML; outputtitle;")
                    set songartist to (execute javascript "var outputartist = document.querySelector('[data-qa=\"mini_track_artist_name\"]').innerHTML; outputartist;")
                    end tell
                    set found_video to true
                    exit repeat
                end if
            end repeat
        end repeat
    end tell
    return songtitle & ":" & songartist
EOF
)

if [ $? -ne 0 ] ; then
    echo "Sorry, No cigar :("
else
    echo $result | awk -F: ' {print "Song: " $1 "\nArtist: " $2  } '
fi

So, now, I get to know the title/artist of the song that is playing on the browser, right in my terminal w/o taking eyes off iTerm!

As we mature, we start realizing the method in the madness — slowly appreciating, how most tasks build on some grammer and we soon realize the proliferation of options isn’t as unwiedly after all, and if we understand the pieces, we can leverage the whole is greater than the sum of parts philosophy with vim.

And eventually, we come across key-mappings in vim — a convenient way to remap some often repeated key-strokes into some shorter keys. So, we get hyper-excited about key mappings and go overboard with mapping most things with shorthands. And then comes the problems :)

• We realize that we masked some existing command with our creative mapping.
• We get trained to our maps, but suddenly they no longer work in another person’s setup or another machine, which doesn’t have our vimrc files, and that is frustrating, especially looking clumsy in a colleague’s desk!

As I was beset with those moments, I stumbled on this vim-blog and specifically on this article. It opened me to thinking anew towards vim-mappings. As mentioned here, the vim-keyscape is pretty saturated. At the outset, there appears hardly any room for new maps without trampling on some existing command. But upon looking through, there are some holes that we can mine. So, I decided, I will list the existing commands and identify the gaps, keep them ready so that I can use those whenever I need a map.

So, I prepared this list over time.

Directly Mappable Hot-Keys

Normal mode commands for each key are really hot — as they are the fastest to type. Over here, I saw that I can reclaim the lowercase s, upper-case Q,S and Z. The substitute(s) closely overlaps with change(c). s is a convenience for changing a single char with many chars, which I rarely used. Further its available with a cl, in case I needed that. Q goes to ex-mode. This was a long-term frustration for me. I would accidentally type Q, and wonder why vim suddenly came over to this ex-mode prompt. And it was a while till I realized, I could come out of it typing a :visual, without having to quit and restart! Finally, I realized, it was the accidental Q, that was causing all the trouble! S is a shorthand for changing current-line. Again, I rarely use that, and cc is a synonym. ZZ is a shorthand for close and exit — not something I do often. So, after listing down all the alphabets and symbols, I mined 4 hot-keys right away. I have mapped them to

• s -> easy-motion-s2
• S -> easy-motion-s2
• Q -> close tab. I go crazy with tabs, especially with vim-fugitive, and need a way to close all those tabs having commits/diffs quickly.
• Z -> :lclose and :lopen location-list. The location window tends to go out of size, when we open new windows and this is a quick way to redraw it.

The g-key

I realized that the g-key has some cute options — with what follows. So, this called for a listing on its own. I listed down all the g-<>commands. The following were revelations

• gd — goto local definition! I had been a c-programmer for so long, and didn’t know that vim has a simple gd out-of-box and we needn’t hop above with # !
• g*/g# — searching current word without word-limiters, handy if we want to later use the / register for some purpose.
• gJ — avoid that annoying space when joining lines!

After listing down all the g-<> commands, there are quite some good holes here. My most used is gw<>. The original gw<motion> will format whatever gw<motion> covers. But I typically visually select my block and press gq. So, gw appeared redundant. I have mapped all window commands to this gw prefix (w for window!). Finally some relief to keep pressing ctrl-w and hurting my little finger.

The next big bang was using gx-<>. It has some netrw usage. But I don’t use netrw and use NERD-Tree plugin. So, i use gx as a cscope-prefix instead of the C-\. More rest to the little finger. gxx is mapped to C-tto unwind back the tag-list.

The next was gh-<>. ghis originally to enter select-mode. But who needs this mode anyway! So, gh-<letters> are mapped to various unite operations.

Listing the various commands

While I still have plenty of mapping-space with the above listing, I realized that I was vimming better by listing the keys in one place — so went ahead and listed the various ctrl- keys in normal mode, ctrl-keys in insert-mode, ctrl- keys in command mode. I found these which I didn’t know till then.

insert-mode ^-o : executes one command and comes back to insert mode. This is handy to move around in insert mode.

insert-mode ^-y/e: type contents of previous/next line. While we can do a yy/p or P and then change what we want, sometimes, if we are in insert mode, this is handy.

So, there, you can start from here and define your maps and relieve your fingers from typing less and hitting ctrl-keys even lesser.

The vanilla commands to interact with cscope are

" Let vim know about a cscope database
:cs add path/to/cscope.out relative/path/to/prepend/to/any/result

" If cscope.out is in the cwd and cscope was build here too, then just
:cs add cscope.out

"Invoke csocpe. goto definition of functionA
:cs find g functionA()

Naturally, it is too tedius to type the :cs find … everytime, we need to lookup something. Not the vim way! There is a plugin cscope_macros.vim to solve just that. It adds the

C-\ g|c|s|f|e

key-maps to respectively do jump-to-definition/callers/all-references at the current keyword. And loads the cscope on startup automatically. This in itself is a great boost to productivity. I am hoping most readers are already doing this much.

One major pain point in this usage is when there are too many results. The normal vim-cscope behavior is to list all results interactively for us to choose with a number. If the result spans more than one screen, vim paginates the results. We skim over the current page, and if our result isn’t shown so far, we move to the next page. We can’t go back to the previous page (unless we are couched inside a tmux/screen window or use the terminal’s scroll-up). Further we painfully note down the number of the result and choose that number. This was largely how I was doing things till I realized vim has a nifty option to load the cscope results into the quick-fix window. Please refer to my previous post if you aren’t familiar with quick-fix/location-list.

This behavior of showing the cscope results interactively or in a quick-fix list is controlled by the following setting

:set cscopequickfix=s-,c-,d-,i-,t-,e-,a-
" or in short:
:set csqf=...

Each of the letters control the action for the respective cscope invocation. The +,-,0 after the letter controls whether the result should be added to an existing list(+) or replace the existing list(-) or 0(don’t use quick-fix and use the interactive option — the default). Absence of the letter in the csqf setting also implies 0. So, setting

:set csqf=s-

makes any cscope-s invocation list all its results in the quickfix. That’s it. There is one problem to this. Vim will automatically jump to the first result. When I initially came across this option in vim, I was excited. But this jumping immediately to the first result, kind of put me off over time, as it broke my flow, by jumping to arbitrary locations that happened to come up first in cscope’s list. So, I turned this off and went back to using the interactive way, until it struck me that I can fix this with a small vimscript. The idea is to mark the current location, set the cscope setting temporarily to load into location-list, invoke cscope, revert cscope setting and jump back to the saved location. This way, the default interactive behavior of cscope is intact which fits for go-to definitions, where results aren’t too many. When on demand, the cscope result can be loaded into location list — and cursor remains wherever we are.

function! LoadCscopeToQuickFix(currword, oper)
  execute "normal mZ"
  execute "set csqf=" . a:oper . "-"
  execute "lcs find " a:oper . " " . a:currword
  execute "lopen"
  execute "wincmd p"
  execute "normal `Z"
  execute "set csqf="
endfunction

nmap <Leader>css         <Esc>:call LoadCscopeToQuickFix(expand("<cword>"),"s")<CR>
nmap <Leader>csg         <Esc>:call LoadCscopeToQuickFix(expand("<cword>"),"g")<CR>
nmap <Leader>csc         <Esc>:call LoadCscopeToQuickFix(expand("<cword>"),"c")<CR>

Here is a small screencast of the above in operation.

Bonus:

I find the cscope’s listing of a C-keyword more useful than a simple grep, as it gives a contextual information — the function in which its present. This helps to quick look for information we need.

Bonus-2:

Use the plugin QFGrep to filter entries from quickfix/location-list.

By far, this was the first time, I came across such a comprehensive analysis. Intuitively, while paying the interest part in a mortgage seems like a drain and this is typically more than the rent when a mortgage starts, as years roll by, the interest part diminishes and the rent keeps growing due to inflation. At this point, its not clear whether renting or buying is better.

I like the way the calculator approaches the comparison. While I wish I got access to the actual comparing code/logic, after reading the explanations on the page, I just came up with my implementation of the same comparison. Mine is not as accurate, as feeding the same inputs gives a different result in mine vs the web page. Probably the actual tax calculations are a bit different. But anyway, my actual desire to (re)implement the calculation was to quickly try out variations on parameters and see how each has a say on the comparison, and manually feeding them on a web-page was turning out cumbersome.

So, here is a list of all factors that are taken in. I am also listing the default values I am assuming. These are baked into the script. We can override any explicitly.

1. Home value ($750,000)
2. How long (30 Yrs)
3. Mortgage (4.0%)
4. Down Payment (10%)
5. Length of Mortgage (30 Yrs)
6. Home Price Appreciation (4.0%)
7. Rent Appreciation (4.5%)
8. Inflation Rate (3.0%)
9. Investment Rate (8.0%)
10. Prop Tax Rate (0.8%)
11. Joint/Single (Joint)
12. Marginal Tax Rate (25%)
13. Buying Expense (1.5%)
14. Selling Expense (6%)
15. Maintenance (1%)
16. Own Insurance (0.46%)
17. Common Monthly Expense (250)
18. Rental Deposit (I am leaving this off)
19. Broker Fees (I am leaving this off)
20. Rent Insurance (0.5%)

For these values, the web-page shows $2199, while my calculation shows $2452. So, my understanding/calcuations err on the side of renting, while the web-site is towards buying. Anyway, even $2452 is not sufficient to rent any $750K house as of now in the bay-area, that I come across. So, overall, for these numbers, the tilt is towards buying. But, lets hold our horses, and study tweaking each parameter.

1. Home Price Appreciation

The most popular push to buying a new home is the fear that home price is getting costlier by the day. But my friends points to this article that pegs the historical home price growth at much less than 4%. So, lets adjust this rate from 2 to 10. Anything above 6, renting doesn’t hold. But below 6, the case for buying isn’t that strong, especially if we tweak the other numbers too. (I pulled up the mortgage value to be same as price_appreciation). The numbers have a decent impact. For example at 3% to 5% the variation is ~$120.

$ for i in $(seq 2.0 0.5 10.0) ; do rent=$(./calculate.py --price_appr $i --down_pay 100 -l 0); echo "$i -- $rent" ; done
2.0 -- 2269.20
2.5 -- 2322.98
3.0 -- 2372.38
3.5 -- 2416.01
4.0 -- 2452.31
4.5 -- 2479.46
5.0 -- 2495.37
5.5 -- 2497.67
6.0 -- 2483.64
6.5 -- 2450.18
7.0 -- 2393.76
7.5 -- 2310.35
8.0 -- 2195.37
8.5 -- 2043.62
9.0 -- 1849.19
9.5 -- 1605.36
10.0 -- 1304.49

2. Mortgage rates

Naturally, more the mortgate rate, the worser is the buying decision. While 4% is our default, an 1% increase here strongly bumps up the break-even rent by $180!

$ for i in $(seq 2.0 0.5 10.0) ; do rent=$(./calculate.py --mort_per $i -l 0); echo "$i -- $rent" ; done
2.0 -- 2126.19
2.5 -- 2203.34
3.0 -- 2283.46
3.5 -- 2366.48
4.0 -- 2452.31
4.5 -- 2540.87
5.0 -- 2632.04
5.5 -- 2725.73
6.0 -- 2821.82
6.5 -- 2920.20
7.0 -- 3020.75
7.5 -- 3123.34
8.0 -- 3227.87
8.5 -- 3334.21
9.0 -- 3442.24
9.5 -- 3551.86
10.0 -- 3662.94

3. Investment rate

Investment rate has to meaningfully hold above 8% to makes any sense. But every percent increase, greatly tilts on renting. Anyway, I dont think, we should assume anything beyond 9% for especially long terms like 30+ years.

$ for i in $(seq 5.0 0.5 10.0) ; do rent=$(./calculate.py --inv_rate $i -l 0); echo "$i -- $rent" ; done
5.0 -- 1636.67
5.5 -- 1786.58
6.0 -- 1930.75
6.5 -- 2069.28
7.0 -- 2202.29
7.5 -- 2329.92
8.0 -- 2452.31
8.5 -- 2569.62
9.0 -- 2682.01
9.5 -- 2789.66
10.0 -- 2892.73

4. House price

This is tricky. Does increasing home price have any effect? So, we will also print how many rents it take to cover up the home-price. It looks like as home-values increase, it looks like buying is better as the break-even rent keeps coming down.

$ for i in $(seq 100000 50000 1000000) ; do rent=$(./calculate.py --home_val $i -l 0); rent_price_ratio=$(python -c "r=($i/$rent);print r"); echo "$i -- $rent -- $rent_price_ratio" ; done
100000 -- 563.37 -- 177.503239434
150000 -- 741.34 -- 202.336309925
200000 -- 914.05 -- 218.806411028
250000 -- 1054.88 -- 236.993781283
300000 -- 1194.63 -- 251.123778911
350000 -- 1334.37 -- 262.29606481
400000 -- 1474.11 -- 271.350170611
450000 -- 1613.85 -- 278.836323078
500000 -- 1753.60 -- 285.127737226
550000 -- 1893.34 -- 290.491934888
600000 -- 2033.08 -- 295.118736105
650000 -- 2172.83 -- 299.14903605
700000 -- 2312.57 -- 302.693540087
750000 -- 2452.31 -- 305.834091122
800000 -- 2592.05 -- 308.636021682
850000 -- 2731.80 -- 311.150157405
900000 -- 2871.54 -- 313.420673228
950000 -- 3011.28 -- 315.480460137
1000000 -- 3151.03 -- 317.356546907

5. Duration of holding the house

The best time seem to be somewhere in the 16 Year’ish — perhaps because of the tax benefits that dip after that. But otherwise, the tenure doesn’t seem to hold much.

$ for i in $(seq 4 4 100) ; do rent=$(./calculate.py --how_long $i -l 0); echo "$i -- $rent" ; done
4 -- 2631.93
8 -- 2264.25
12 -- 2232.43
16 -- 2272.13
20 -- 2330.52
24 -- 2384.35
28 -- 2431.42
32 -- 2498.54
36 -- 2565.80
40 -- 2609.67
44 -- 2638.16
48 -- 2656.35
52 -- 2667.55
56 -- 2673.95
60 -- 2677.07
64 -- 2677.95
68 -- 2677.30
72 -- 2675.64
76 -- 2673.34
80 -- 2670.64
84 -- 2667.73
88 -- 2664.75
92 -- 2661.77
96 -- 2658.87
100 -- 2656.07

6. DownPayment

Interestingly, I was wondering the case, where we buy the house outright w/o mortgage. Guess, what it costs $3727 vs the $2452! So, it makes sense to mortgage, espeically when we can invest money in hand in stock-market and not pre-pay anything! Thank you tax-exemptions.

$ for i in $(seq 10 10 100) ; do rent=$(./calculate.py --down_pay $i -l 0); echo "$i -- $rent" ; done
10 -- 2452.31
20 -- 2599.03
30 -- 2745.74
40 -- 2892.45
50 -- 3039.17
60 -- 3185.88
70 -- 3332.60
80 -- 3479.31
90 -- 3618.21
100 -- 3727.20

7. Common Monthly Expenses

There are the community charges etc.. that we need to pay. This has the highest say — as it almost translates directly to rent.

$ for i in $(seq 0 100 1000) ; do rent=$(./calculate.py --month_comm $i -l 0); echo "$i -- $rent" ; done
0 -- 2244.87
100 -- 2327.85
200 -- 2410.82
300 -- 2493.80
400 -- 2576.78
500 -- 2659.75
600 -- 2742.73
700 -- 2825.70
800 -- 2908.68
900 -- 2991.65
1000 -- 3074.63

Just avoid houses with these charges :) A better deal would be to chose a costlier house without it. For example, its almost same in break-even rent (~$2450)to choose a home at $825K with no monthly charges to a house of $750K with a $250 monthly charge.

8. Property Taxes

Again, more the taxes, more the loss (No, tax exemptions dont mean much)

$ for i in $(seq 0.5 0.1 1.5) ; do rent=$(./calculate.py --prop_tax $i -l 0); echo "$i -- $rent" ; done
0.5 -- 2320.78
0.6 -- 2364.63
0.7 -- 2408.47
0.8 -- 2452.31
0.9 -- 2496.16
1.0 -- 2540.00
1.1 -- 2583.84
1.2 -- 2627.69
1.3 -- 2671.53
1.4 -- 2715.37
1.5 -- 2759.22

9. Other parameters

There are hardly any change in the following. So, I am not tweaking this. Maintenance might be more or less. Since we are accounting compounded maintenance, I guess, the 1% assumption fairly evens out this expense.

• Joint/Single, assumed joint
• Marginal Tax Rate (25%)
• Maintenance (1%)
• Own Insurance (0.46%)
• Rent Insurance (0.5%)

These parameters dont make much impact.

• Buying Expense (1.5%)
• Selling Expense( 6%)

BottomLine

One thing I realize is, depending on our assumptions on the 3 big-ticket items — Home-value appreciation, Rent appreciation and Investment Returns, our decision will change. None of them are easy to arrive it. While there are the broad indices and past-performances, the exact house we look at or the investment we make, might differ from these broad averages. I ball-park the values at 4,4,9. So, I tend to decide towards renting. This is further emboldened by the fact that I don’t have money anyway to pay for a downpayment!

The biggest take-away for me over this exercise is that, neither buying or renting, is unquestionably better than the other. While we may not have to shy away from buying altogether, we needn’t think buying is the must-to-do thing either. The house price/rent-appreciation are definitely not as high as investment-returns. So, participating in competitive bids to just buy a house early in our career, is totally unnecessary — renting is not loss of money, and house prices and rents aren’t beating general inflation as much as investment-returns.

Further, while we buy, we typically over-shoot for a slightly bigger house than we would otherwise rent. Also we tend to pick on considerations that have temporal needs, like proximity to work or school-districts, which we can move away from, in case of renting.

Happy renting, buying or well, deciding :)

"The window opened by vim on these commands is called the QuickFix
:copen
:cnext
:cprev
:cc
:cf <file>
:cclose

In addition to quickly moving around compile-results, we can load grep results in vim and move around with exactly the same commands. If we think about it, both compilation errors and grep results have similar patterns. Each entry has a file-name, line-number to go to and a string to watch out for — the error message in case of a compilation-result or the line itself in case of grep. To open a grep result in quick-fix:

"This will invoke an external grep
:grep pattern

"Or you can do grep externally in a terminal and collect 
"its result in a file.
"Ensure to do grep -nH to get filename and line numbers
:cfile file_having_grep_results

"Note that vim jumps to first result right on :cf. 
"To view the results
:copen

Location List Vs QuickFix

QuickFix window is global to one vim-session. If we have lots of windows, they all share the same quick-fix. If we are wading over compilation errors, one global quickfix window is sufficient — as we compile per project and there aren’t typically different compilation error lists. However once we start using grep, we soon run into not wanting to disturb the grep results loaded in one window and still do another grep in another window. In such cases, we can use the location list. The location list is exactly same as quickfix, just that its private to each window! The commands for the location list are very similar. Just substitute l instead of c.

:copen      :lopen
:cnext      :lnext
:cprev      :prev
:cc         :ll
:cf <file>  :lf <file>

In fact, we dont have to grep separately in a terminal, but rather grep from within vim. There are some nice plugins that grep every efficiently in source-repositories. I use Ag.nvim. One very common usecase for me is to grep in the current file quickly. Vim offers that out-of-box without any plugin. Sometimes, its convenient to view all results right in one place, instead of having to hop over with n’s.

"Load into quickfix. vimgrep is vim itself doing a grep, 
"  instead of using a external grep
"This is okay in our case, as the current file is already a 
"  buffer in vim's memory
:vimgrep /pattern/ %

"Or if we want in location window
:lvimgrep /pattern/ %

As is common to any vim work-flow, the sum is greater than parts. Note that the quick-fix and location-lists are almost vim-windows on their own right. So we can vim-around(eg, yank or search/highlight text) within the results in the quick-fix/location-list! After using it for sometime, we eventually want to grep within the grepped-items, as we start from a broader to narrower search. To this effect, there is a nice plugin — QFGrep, which further filters items on quickfix! (The quickfix is non-modifiable buffer, so a simple v/pattern/d wont work here. This plugin kind of does just that!)

Here is a screencast illustrating, lvimgrep/QFGrep in action.

In a subsequent blog, I will show, how cscope can be combined with this QuickFix/Location-List for efficient code-browsing.

I knew mac is popular with the developer community for its swiftness and coolness and was wondering how I would get on with it. I am a very keyboard shortcut’y person and spend most of work in a terminal. I was using auto-hotkey enriched with an application-key-turned-capslock setup, as suggested in this blog, and would do most mouse’y stuff like scrolling, accessing context-menu, from my home-row position, and was wondering If I can live by with same keyboard speeds on the mac.

After two months of using mac, I am totally in love with it. I thought I will share a summary of how I find it so far.

AWESOME in mac

• weight of the laptop

Honestly, this is nothing to do with mac per-se. But, all windows laptops I used/owned before were heavy bricks — so, mac is my first feather-weight machine.

• Screen Resolution

Again, not exactly mac’y and there are good resolution windows machines. But, (I have a 4K monitor driven by windows-10 laptop), the ease with which Mac adjusts to the high resolution display truly tops. In windows, now and then, I will have to rescale/adjust apps. Not sure about the native high-resolution screen laptops though.

• Speed with which its ready to work on — both sleep/boot

The feature that windows can never come close. Chromebooks are on par though ;) In fact, in my windows laptop, I still can’t figure out, why the Dell-bios-like screen splashes sometimes for seconds on wakeup and sometimes doesn’t.

• its full-screen mode, exactly how I want it. We still get system notifications popped up and when we hover mouse up, we get access to the application menu and system status icons, instead of having to exit full-screen first!
• terminal is a first class citizen, unlike in windows, where I use a terminal off a virtual-box’ed linux!
• iterm integration with opening files by “open”.

Awesome, but I don’t really use that much

• mouse gestures
• The splash screen of all apps on 3-finger-scroll or ctrl-up. I have hotkeys for all apps I frequently use. But once in a while, I do use this to get stock of my desktop.

small cute things in mac

• native ability to map caplock (I mapped to command)
• native ability to O(1) switch to another app by typing its name — spotlight. I although have changed to quick-silver. Further, typing the app, just opens the running instance, instead of spawning a new one, like the win-r run window in windows!
• I used work-rave in windows, to remind me of breaks. At its timer, it would just grab keyboard focus, spoiling what I am typing [perhaps intended :) ]. I now use time-out in mac, which gracefully dims up the screen w/o grabbing keyboard/mouse strokes.
• Amphetamine (switched to this after using caffeine for some time), which allows to not sleep. Very handy, when I am on discussions/meetings.

Bummers in that order:

• Outlook has no format painter! [ :( ] Yikes!
• Auto-hotkey stuff (karabiner seems unavailable in mac-os sierra, that is suggested as alternative). I basically just need scrollup/down, and hjkl map for arrow-keys which I am so used to, in windows.

Other minor inconveniences:

• I would like to change the lock-up screen from the blurred image to a proper image — com.apple.desktop.admin.png solution keeps reverting after some time!
• Absence of a application-key that emulates a right click at a location.
• Having to unlearn and relearn key-sequences. Especially frustrating is typing end-key hoping cursor to come to end of line, but finding it going to end of page. And having to painstaking come back to the current line, when on web-forms. And word-jumping is alt-arrow and not ctrl-arrow.

Issues that were bothering me, but found a way out.

• mission control should have a way to keyboard-select app of choice.
   — Just use spotlight/quick-silver.
• Hotkey to open up apps. I was doing it the apple-script/services way. Hotkey to lock too has same flaky behavior
   — Quicksilver’s trigger is now how I assign global-hotkeys to apps.
• How to auto adjust font-size for iTerm based on display?
   — I am just manually cmd- +/-’ing on demand.
• how to move all windows to another display in dual monitor case, when i haven’t closed my mac-lid
   — spectacle.app solved this by offering the shortcut to move windows across monitors, although one by one, which is good enough, as long as I don’t have to reach for the mouse.
• zoom/maximize sometimes only vertically maximizes (eg: iTerm, chrome!, no option/shift/command click doesn’t work either)
   — I found spectacle.app, which does the job now (and with a keyboard shortcut too!)
• Shut display of mac with a second monitor, with mac lid open
   — Still can’t. But since I found spectacle.app, I don’t mind keeping mac open and having a few windows there.
• One thing I realized after coming out of windows, is how handy the windows calendar is, to pop up from system-tray! Fortunately mac has a itsyscal app that does this.

ptr_typeA->memberBptr->memberC;

To complicate things, member names like id, for example, can be used across many structures and doing a grep or even (cscope find s) on such identifiers will throw gazillion results. To correctly go to definition on such instances or list all references, we need a compiler-class source code browsing, and cscope isn’t that. I am not very familiar with clang and its internals, but clang offers more than just compilation — it emits source-code information so that the above is possible.

After some reading and experimentation here and there, I finally found the following working well for me

1. Deoplete-clang, for code-completion
2. rtags, for code-browsing. rtags was primarily developed for emacs, but a vim-frontend is available!

Both of them are dependent on clang. The following site has excellent steps to build clang. clang in turn requires a modern gcc. With the dockered centos, I was able to follow the instructions to the dot and get clang compiled. The results were available in /usr/local/ of the container. I just tar-balled the entire /usr/local and scp’ed it to my dev machine’s $HOME/install/clang folder and untarred there. The $HOME/install/clang/lib/libclang.so had all its dependencies met. In fact, the binaries in $HOME/install/clang/bin too had no trouble finding the libraries from the nearby lib folder. Hurray!

Both the deoplete-clang and vim-rtags plugin had options to point to the clang library, rc executables via vimrc variables. I updated my PATH and LD_LIBRARY_PATH to pick the clang and other executables. And fortunately rtags has a cute option of wrapping gcc to generate the clang-compilations and that just worked.

And presto!, my code-completion and source-code browsing is now at a new-level :)

I thought it would be just doing a

wget …mosh..
tar xf mosh..tar..z
./configure — prefix=$HOME/install
make
make instal

Unfortunately, mosh requires the following development packages:
1. perl
2. protocol buffers
3. ncurses
4. zlib
5. open-ssl

Each of them in turn kept depending on something else, and pretty soon, I lost track of what I am compiling, the CXXFLAGS/LD_LIBRARY_PATH I seem to be setting and installing. I just gave up after something complained that the gcc version I had in my machine is too old. I had almost given up on using mosh in my office setup, until I realized I can try installing mosh in a docker centos and finally cp’ing the results back to my machine.

So, went to docker repository and then realized, even docker doesn’t have a centos 6.5 image and starts only with centos 6.6. Hrmm. Anyway, trusted my luck with centos 6.6 and sure enough, with root powers, quickly yum installed most dependencies. I had to compile-from-source just the proto-buf library and then mosh. Mosh finally compiled and mosh seems to depend on these libraries:

[root@31837a315921 sourceInstallations]# ldd mosh_build/bin/mosh-client
 linux-vdso.so.1 => (0x00007fff397e6000)
 libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007fb47eac5000)
 libprotobuf.so.12 => /usr/local/lib/libprotobuf.so.12 (0x00007fb47e688000)
 libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007fb47e41c000)
 libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007fb47e037000)
 librt.so.1 => /lib64/librt.so.1 (0x00007fb47de2e000)
 libz.so.1 => /lib64/libz.so.1 (0x00007fb47dc18000)
 libstdc++.so.6 => /usr/local/lib64/libstdc++.so.6 (0x00007fb47d88b000)
 libm.so.6 => /lib64/libm.so.6 (0x00007fb47d606000)
 libgcc_s.so.1 => /usr/local/lib64/libgcc_s.so.1 (0x00007fb47d3f0000)
 libc.so.6 => /lib64/libc.so.6 (0x00007fb47d05c000)
 libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb47ce3e000)
 /lib64/ld-linux-x86–64.so.2 (0x000055ab177b4000)
 libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fb47cbfa000)
 libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fb47c913000)
 libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fb47c70e000)
 libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fb47c4e2000)
 libdl.so.2 => /lib64/libdl.so.2 (0x00007fb47c2de000)
 libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fb47c0d2000)
 libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fb47becf000)
 libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fb47bcb5000)
 libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fb47ba95000)

I decided I will start with just cp’ing the mosh binaries alone and then cp the dependent so-files one by one. Fortunately, most libraries were there in my development machine (so, much for all the missing dev-pkg dependencies to compile!), except for the protobuf. Once I cp’ed the libprotobuf.so.12 and set up LD_LIBRARY_PATH in my bashrc to its location, voila!, mosh just worked.

Yippee! Thanks Docker.