Blaz Merela

Rails boot sequence

Thu, 19 Jan 2023 00:00:00 GMT

As I was having issues with a misbehaving gem, I needed to demystify Rails’ boot sequence for myself. As a (brief) web search also didn’t return many visual resources, I made a summary of Xavier Noria’s talk.

To summarize in words:

Define load paths for deps (Bundle.setup)
Load rails/all (railties of Rails components)
Load gem deps (Bundle.require)
Define the Application class
Define assorted paths (autoload etc.) #Rails.application.initialize!
Load config/environments/#{Rails.env}.rb
Load initializers in config/initializers
Eager load application (if nil). Happens by default in production mode.
Load config/routes.rb
Run command hooks (if needed).

Testing IPv6 connectivity on AWS

Sat, 05 Nov 2022 00:00:00 GMT

Do you need to verify that a particular service handles IPv6 traffic, yet your ISP doesn’t provide you an IPv6 address? This guide will help you work around this by setting up an IPv6-only EC2 and connecting it to the service in question.

As a rough overview, you’ll need at least these pieces:

dual-stack VPC
dual-stack ALB
A(AAA) record to the ALB
IPv6-only subnet
IPv6-only EC2

We’ll highlight the needed modifications for each, using Terraform. There’re more pieces needed, but the unlisted ones aren’t directly related to IPv6.

Dual-stack VPC

We start with an already existing VPC. The following configurations have to be added to the VPC module:

enable_ipv6 = true
assign_ipv6_address_on_creation
private_subnet_assign_ipv6_address_on_creation
public_subnet_ipv6_prefixes
private_subnet_ipv6_prefixes
database_subnet_ipv6_prefixes

enable_ipv6 lets Amazon provide you an IPv6 CIDR block, in which case you cannot set on its prefix length nor IP range. (You may alternatively bring your own IP.)

IPv6-only subnet

The subnet’s definition is straightforward:

resource "aws_subnet" "this" {
  vpc_id                                         = var.vpc_id
  ipv6_cidr_block                                = var.cidr
  ipv6_native                                    = true
  assign_ipv6_address_on_creation                = true
  enable_resource_name_dns_aaaa_record_on_launch = true
}

Here’s how you might set the above vars. (We’re using Terragrunt, thence the dependency keyword.)

locals {
  # Fixed when using AWS-provided IPv6 CIDR block.
  cidr_prefix_length_vpc    = 56  cidr_prefix_length_subnet = 64  count_existing_subnets    = 8}
vpc_id = dependency.vpc.outputs.id
cidr   = cidrsubnet(
  dependency.vpc.outputs.ipv6_cidr,
  local.cidr_prefix_length_subnet - local.cidr_prefix_length_vpc,
  local.count_existing_subnets + 1)

Two things to note about the highlighted lines, in order:

The CIDR prefix lengths are fixed: at 56 for VPCs and at 64 for subnets. Hence the VPC’s CIDR prefix is extended by 64 - 56 bits above.
The VPC already had 8 existing subnets and we add the IPv6-only subnet as the 9th one; thus the 9 on the 2nd highlighted line above.

(Note: outside of this hypothetical scenario, this particular IPv6 subnet would be provisioned from the same configuration as the rest. That is, there wouldn’t be a + 1 subnet added in such an ad hoc manner.)

ALB

ip_address_type = "dualstack"

Related to ALB, note that, as of now, its Target Group must be of target_type = "ip".

IPv6 target groups only support IP type targets.

A(AAA) record

The AAAA record is required for our IPv6 testing, while the A record is a nice to have, as it’ll make a potential Route53 healthcheck pass. If using Route53, it’s best to use an ‘alias’ extension, letting the AAAA record resolve via ALB’s DNS.

EC2

Only changes to the "aws_instance" resource are

the addition of ipv6_address_count, set to a non-zero value, and
disablement of associate_public_ip_address which presumes an IPv4 address. Then, as long as you pick a relatively recent ami, you should be all set.

In one the dual-stack subnets you’ll also need an EC2 with both IPv4 and IPv6 connectivity.

Conclusion

Finally, to actually test the above setup, you’d SSH into a dual-stack EC2 and from there on SSH further into the IPv6-only EC2. From there you’d then, finally, be able test the IPv6 connectivity of your service in question.

Note that you could also merely add an IPv6 interface directly to the service’s EC2 and obviate the need of setting up the LB. But since many services find themselves behind an LB, I found it fitting to include ALB setup as well. Especially since its target-groups currently have an IPv6-related limitation.

We haven’t laid out all the steps in detail, though the above are the important bits needed to test IPv6 in AWS. Reach out if you aren’t able to fill in the missing gaps and I’ll expand on the unclear parts.

Safer terminal

Tue, 19 Jul 2022 00:00:00 GMT

As much as we strive towards having the whole server fleet be treated as an indistinguishable herd of cattle, there are still times when taking care of pet-servers is necessary. One such time is when the infrastructure is still undergoing a cloud migration, or when a particular server really is a special darling.

In such cases, any additional feedback to the operator helps in avoiding mistakes. In this post you’ll see approaches that you might use to enable such visual feedback. The end goal is to immediatelly visually discern that one has entered a potentially dangerous environment, one where any mistake might be of grave consequences.

Most of the time, an operator’s tool of choice is a terminal. We’ll focus on iTerm2, a terminal on macOS. But even if you’re not an iTerm2 user, the following tips might inspire you to apply something similar in your chosen terminal.

iTerm2

Triggers

The prerequisite for Triggers is to ‘Install Shell Integration’. You’ll find it in iTerm’s menu bar:

Then, you can enable Automatic Profile Switching. But, since installing APS on remote hosts is often impractical or inadvisable, you can work around this by creating a so-called Trigger.

First create a new profile, say “Production”, and under “Advanced” tab add a Trigger with the following configs:

Regular Expression: ^(\w+)@([\w.-]+):.+\$ (amend if necessary)
Action: Report User & Host
Parameters: \1@\2
Instant: checked
Enabled: checked

Then head back to the APS setting and insert hostnames of your production servers. Finally, for this new profile set a distinct visual feedback under the “Colors” tab; white-on-red would be fitting.

Badges

Another, or additional, option is to set a badge, per iTerm pane, when a particular command gets executed. You can wrap the existing command for switching your infrastructure-related profiles and have the wrapper also execute iTerm’s SetBadgeFormat command. Here’s a sample:

declare -a important_profiles=("prod" "stag")
if [[ "${important_profiles[*]}" =~ "$profile" ]]; then
  echo -e "\x1B]1337;SetBadgeFormat=$(echo "Careful: ${profile}" | base64)\007"
fi

Regex matching on longer lines

In case the part of the log that you want to match on has (too) many logical lines, you should increase the following iTerm limit: Preferences > Advanced > Number of screen lines to match against trigger regular expressions.

Safer shell sub-commands

Mon, 13 Jun 2022 00:00:00 GMT

You will undoubtedly encounter deprecated, inadvisable, or outright dangerous commands in your daily work. Rather than having to rely on your erring, fallible memory, it would be better if you could guarantee avoidance of such commands. Or at least have them behind an additional prompt. This is especially important in infrastructure work, where the stakes are that much higher.

You might be reading the documentation on terraform refresh and come across the following warning:

Automatically applying the effect of a refresh is risky. If you have misconfigured credentials for one or more providers, Terraform may be misled into thinking that all of the managed objects have been deleted, causing it to remove all of the tracked objects without any confirmation prompt.

Rather than form a memory of the quoted bits in your hippocampus, you’d be better off storing these bits into your appropriate ~/.{bash,zsh,etc}rc runcommand file:

terraform() {
  case $1 in
    refresh)
      terraform apply -refresh-only
      ;;
    *)
      command terraform "$@"
      ;;
  esac
}

You now have a guarantee that the alternative, recommended subcommand apply -refresh-only will be used. This pattern can be applied on other dangerous and/or deprecated subcommands, be it for terraform or some other command. How ~/.gitconfig allows for customized subcommands might also serve you as a further inspiration.

Packer 101

Mon, 06 Jun 2022 00:00:00 GMT

I was in the midst of modernizing client’s existing AWS infrastructure, when the client requested that one of their core server-cluster be upgraded.

Since a full migration onto a container orchestrator was months in the future, I still deemed it wortwhile to make at least some improvements, in the form of custom AWS AMIs. As the development team was relatively small, my goal was to lessen their need to fiddle with the servers, be it with SSHing onto it or needing to manually set up the needed monitoring tools. I also provided a fully automated deployment so that running an Ansible playbook wasn’t necessary anymore, at least for these servers.

Show me the code!

For the actual implementation, the prebaked VM needed

a CodeDeploy agent, for automated deployments, and
a Prometheus node-exporter, given the planned EKS migration. We used Packer as our tool of choice. (If you’re fully committed to AWS you can also use ‘EC2 Image Builder’.)

Presuming you have asdf installed, the zeroth step is to get Packer:

asdf plugin-add packer https://github.com/asdf-community/asdf-hashicorp.git
asdf install packer 1.8.0
asdf local packer 1.8.0

In the same directory, create an empty file with the .pkr.hcl extension. We’ll add the required packer, source, and build blocks to this file in the following paragraphs.

In packer block, pin to the same version you just installed.

packer {
  required_version = "~> 1.8.0"
}

For easier maintenance, extract the important configs into locals:

locals {
  aws_region = "eu-west-1"
  instance_type          = "t3a.micro"
  # If more than a single such AMI needs to be built each day, make the AMI's time-signature more granular.
  today_date = formatdate("YYYY-MM-DD", timestamp())
  ami_owner_id_Canonical = "099720109477"
  ami_ubuntu_version     = "focal-20.04"
  ami_ubuntu_arch        = "amd64"
  ami_name   = "ubuntu-${local.ami_ubuntu_version}-${local.ami_ubuntu_arch}-${local.today_date}"
}

The source block is pretty much standard. Setting a few well chosen tags will come helpful later, such as when provisioning servers of mixed architectures.

source "amazon-ebs" "custom" {
  ssh_username          = "ubuntu"
  region                = local.aws_region
  instance_type         = local.instance_type
  ami_name              = local.ami_name
  force_delete_snapshot = true
  launch_block_device_mappings {
    delete_on_termination = true
    device_name           = "/dev/sda1"
    encrypted             = false
    iops                  = 3000
    throughput            = 125
    volume_size           = 8
    volume_type           = "gp3"
  }
  source_ami_filter {
    filters = {
      name                = "ubuntu/images/*ubuntu-${local.ami_ubuntu_version}-${local.ami_ubuntu_arch}-server-*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = [local.ami_owner_id_Canonical]
  }

  tags = {
    Name          = local.ami_name
    ManagedBy     = "Packer ${packer.version}"
    Base_AMI_ID   = "{{ .SourceAMI }}"
    Base_AMI_Name = "{{ .SourceAMIName }}"
  }
}

Finally the build block. This one contains two shell provisioners, and an Ansible playbook that installs an AWS CodeDeploy agent. Note that a provisioner is named from the POV of the provisioned server: a -local provisioner, such as ansible-local, runs on the server itself.

build {
  sources = ["source.amazon-ebs.custom"]
  provisioner "shell" {
    execute_command = "echo 'packer' | sudo -S env {{ .Vars }} {{ .Path }}"
    script          = "${path.root}/scripts/base.sh"
    max_retries     = 1
    timeout         = "5m"
  }
  provisioner "shell" {
    execute_command = "echo 'packer' | sudo -S env {{ .Vars }} {{ .Path }}"
    script          = "${path.root}/scripts/prometheus_node_exporter.sh"
    max_retries     = 1
    timeout         = "5m"
  }

  provisioner "ansible" {
    playbook_file = "${path.root}/scripts/codedeploy.yml"
    extra_arguments = [
      "--extra-vars",
      "aws_region=${local.aws_region}"
    ]
  }

}

Supporting scripts

Per above .hcl, I’ve placed Bash and Ansible into a sibling scripts directory.

The base.sh provides unattended APT upgrades:

#!/usr/bin/env bash
set -Eeu
export DEBIAN_FRONTEND="noninteractive"

apt_conf_periodic="/etc/apt/apt.conf.d/10periodic"
apt_conf_unattended_upgrades="/etc/apt/apt.conf.d/50unattended-upgrades"

cat << END > "$apt_conf_periodic"
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
END

cat << END > "$apt_conf_unattended_upgrades"
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}";
    "${distro_id}:${distro_codename}-security";
    "${distro_id}ESM:${distro_codename}";
};
Unattended-Upgrade::Package-Blacklist {
};
Unattended-Upgrade::DevRelease "false";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
END

chown -R root:root "$apt_conf_periodic"
chown -R root:root "$apt_conf_unattended_upgrades"

service unattended-upgrades restart

prometheus_node_exporter.sh sets up and starts Prometheus’ node/host exporter:

#!/usr/bin/env bash
set -Eeuo pipefail
export DEBIAN_FRONTEND="noninteractive"

###################################
# Config
node_exporter_arch='linux-amd64'
node_exporter_version='1.3.1'
###################################

new_user=prometheus
sudo useradd --no-create-home --shell /bin/false "$new_user"


tar_file="node_exporter-${node_exporter_version}.${node_exporter_arch}.tar.gz"

wget "https://github.com/prometheus/node_exporter/releases/download/v${node_exporter_version}/${tar_file}"

tar xvfz "${tar_file}"

base_dir="${tar_file%.tar.gz}"

dir="/etc/prometheus"
mkdir -p "$dir"
mv "$base_dir/node_exporter" "$dir"
rm "$tar_file"
chown -R "$new_user":"$new_user" "$dir"

cat << END > /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
ExecStart=/etc/prometheus/node_exporter
[Install]
WantedBy=default.target
END

systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter

And codedeploy.yml is an Ansible playbook that install a CodeDeploy agent:

---
- hosts: all
  user: ubuntu
  become: yes
  become_user: root
  become_method: sudo
  tasks:

    - name: Install Dependencies
      apt:
        pkg: ['ruby-full', 'wget']
        state: present
        update_cache: true

    - name: Gather package facts
      package_facts:
        manager: auto

    - name: Install Code Deploy Agent
      block:
        - name: Fetch the CodeDeploy install script
          get_url:
            url: "https://aws-codedeploy-{{ aws_region }}.s3.{{ aws_region }}.amazonaws.com/latest/install"
            dest: /tmp/codedeploy-install
            mode: 0700

        - name: Run the installation script
          become: true
          command: /tmp/codedeploy-install auto
      when: "'codedeploy-agent' not in ansible_facts.packages"

Once you have all the above in place, building your AMI is merely a matter of:

input='a.hcl'
packer fmt $input \
&& packer validate $input \
&& packer build $input

You can manually verify that the desired services were indeed provisioned. Instantiate an EC2 off of the newly minted AMI and run:

service unattended-upgrades status
service codedeploy-agent status

Closing tips

The most general tip is to enable debug logs when you encounter an issue, using PACKER_LOG=1 before a Packer command. More specific tips are below, each in its own section.

“Builds finished but no artifacts were created.”

If you encounter this error, recheck whether a source actually is being referred to in your build block:

source "amazon-ebs" "custom" {  ...
}

build {
  sources = ["source.amazon-ebs.custom"]  ...

Packer console

If you wish to use packer console for anything more than mere variable evaluation, such as

echo {{timestamp}} | packer console

you have to start it with the --config-type=hcl2 flag. Only then will you also be able to evaluate HCL expressions:

> formatdate("YYYY-MM-DD", timestamp())
> 2022-04-24

Ansible and “unrecognized arguments”

You might encounter this in the logs, leading to Packer exiting.

amazon-ebs.custom: ansible-playbook: error: unrecognized arguments: /path/to/codedeploy.yml

If Ansible gives you trouble, verify that you’re quoting extra-vars correctly; per docs, “arguments should not be quoted”.

- not OK
extra_arguments = [
  "--extra-vars \"foo=${local.bar}\""
]

+ OK
extra_arguments = [
  "--extra-vars", "foo=${local.bar}"
]

Next up

In other articles, we’ll see how we can make use of the now-baked-in node-exporter for monitoring with Prometheus. And how we can leverage the AWS CodeDeploy agent for automated deployments onto EC2 instances.

Scylla migration

Tue, 01 Feb 2022 00:00:00 GMT

Scylla is a Apache Cassandra-compatible NoSQL data store, with a significantly improved performance. I have recently successfully migrated a self-managed Scylla cluster to a managed, Cloud Scylla. Here are a few takeaways.

First, from a business perspective, the migration made sense as the team was too small to justify spending further engineering time in order to (safely) keep up with the required upgrades. The business had predictable usage patterns, but couldn’t afford any downtime; and the data was large enough that the estimated downtime would’ve been measured in hours. So we chose to migrate using a transitional period of dual-writes, a fork & lift operation for the older data with scylla-migrator, and finally kept the old cluster for a verification period of dual-reads.

Meanwhile I also updated the services using Scylla and have them autodiscover the nodes, courtesy of a particular client-side library. On the cluster side, I also enacted RBAC for both 1) backend services and 2) QA & developer team. Whereas previously they all shared an unlimited superuser role, now they each have their own role, with least privileges necessary:

For backend services:

create role app1 with password = 'app1' and login = true;
create role app2 with password = 'app2' and login = true;

-- Only grant the needed rights to each app.
grant select on keyspace akeyspace to app1;
grant select on keyspace akeyspace to app2;
grant modify on akeyspace.atable   to app2;

For QA team and developers:

-- Implies `SUPERUSER = false AND LOGIN = false`.
create role qa;
grant select on keyspace akeyspace to qa;
grant modify on akeyspace.atable to qa;

create role alice with password = 'alice' and login = true;
grant qa to alice;

create role readonly;
grant select on keyspace akeyspace to qa;

create role bob with password = 'bob' and login = true;
grant readonly to bob;

Thus, no more risking that a fatfingered QA team member, or a developer indirectly through one of the apps, would execute a DROP, ALTER, or other dangerous command. Unfortunately, MODIFY cannot be made more granular so as to allow a DELETE but not a TRUNCATE. This limitation is unlikely to disappear, since Cassandra-compatibility must be kept intact.

A nice touch was to also enable workload prioritization, available only with Scylla Enterprise:

CREATE SERVICE_LEVEL olap WITH SHARES = 100;
CREATE SERVICE_LEVEL oltp WITH SHARES = 1000;

ATTACH SERVICE_LEVEL olap TO qa;
ATTACH SERVICE_LEVEL olap TO readonly;
ATTACH SERVICE_LEVEL oltp TO app1;
ATTACH SERVICE_LEVEL oltp TO app2;

Finally, to review the changes made:

LIST ALL PERMISSIONS [OF <role>];
LIST ALL ATTACHED SERVICE_LEVELS;

Disable Mac App Store notifications

Sun, 16 Jan 2022 00:00:00 GMT

Since App Store isn’t listed under System Preferences > Notifications, here’s a workaround that indirectly disables its notifications by setting the “last” update far into the future.

defaults write \
  com.apple.appstored LastUpdateNotification \
  -date "2099-12-12 12:00:00 +0000"

RabbitMQ, OAuth2, and ConduitAMQP

Sun, 14 Nov 2021 00:00:00 GMT

amqp is one of the most used RabbitMQ Elixir libraries used by many projects directly and also a dependency of Broadway RabbitMQ. It does its job well, but it lacks few polishing touches that are usually needed in a ‘real world’ usage. F.e. you’re on your own when it comes to handling connection hickups. Also, its README presumes at least somewhat experienced user. One knowing that it wouldn’t be OK to merely spawn a “bare” process for the consume/4, as it was suggested in an older README version. But also a user that knows that even the proposed Task should often rather be a supervised one, using Task.Supervisor.

I have already equipped amqp with connection monitoring in previous projects, using :gen_statem behaviour and Process.monitor(conn.pid). It worked, and it also allowed the whole service to boot even if the AMQP broker was unreachable at that moment. But :gen_statem implementation was bit verbose, so this time I picked ConduitAMQP for a new project. The library helpfuly builds on top of AMQP and leverages Connection to provide resiliency.

An issue encountered

Though, as this project required an AMQP connection to be authenticated via OAuth2, and as often more code means more bugs, I hit upon an issue where ConduitAMQP library wasn’t relaying connection options to its amqp dependency, as summarized in the following image.

So I got blocked for quite some time, by not being able to authenticate against RabbitMQ, even though the Keycloak-vended JWT had all the expected fields set. Luckily, succesfully using the same token against RabbitMQ’s management/HTTP API was a hint that the issue might lie with the ConduitAMQP library itself.

First I checked RabbitMQ’s own logs, but they were a bit sparse

HTTP access denied: Authentication using an OAuth 2/JWT token failed: provided token is invalid.

giving no details of the root cause of invalidity.

Going down the RabbitMQ hole

So I was left with no other choice but to amend and build RabbitMQ from its source code. First I exposed Err to the logs, making the following change in rabbit_auth_backend_oauth2.erl,

@@ -85,8 +85,8 @@ update_state(AuthUser, NewToken) ->
  case check_token(NewToken) of
    %% avoid logging the token
    {error, _} = E  -> E;
-     {refused, {error, {invalid_token, error, _Err, _Stacktrace}}} ->
-       {refused, "Authentication using an OAuth 2/JWT token failed: provided token is invalid"};
+     {refused, {error, {invalid_token, error, Err, _Stacktrace}}} ->+       {refused, "Authentication using an OAuth 2/JWT token failed: provided token is invalid: ~p", [Err]};    {refused, Err} ->
       {refused, rabbit_misc:format("Authentication using an OAuth 2/JWT token failed: ~p", [Err])};
    {ok, DecodedToken} ->

then set the relevant config in a .config file,

[
  {rabbit, [
    % Other settings, f.e.
    % {log_levels, [ ... ]},
    {log, [      {console, [        {level, debug},        {enabled, true}      ]}    ]}  ]}
].

and finally ran make run-broker RABBITMQ_CONFIG_FILE=above.config. Thanks to the above config, RabbitMQ’s console now finally clued me in to the actual cause of token invalidity

HTTP access denied: Authentication using an OAuth 2/JWT token failed: provided token is invalid: {badarg, [<<"guest">>]}

which made me realize that Conduit AMQP likely wasn’t respecting the provided options, username: "" and password: token, and thus making the underlying amqp library falling back to "guest" as the default username and password. Actually, the culprit was me, at least initially, as I missed the documentation’s nuanced point of exclusivity between the :url and other options; I’ve since emphasized this exclusivity for future users. But even better, I’ve later also realized that amqp library supports both URI and options since 1.3.0, I’ve updated ConduitAMQP to make use of this.

(Should you also want to run RabbitMQ from its source, I’ve had success with these asdf-installed tools:

make   4.3
bazel  0.13.0
erlang 24.0.5
elixir 1.12.2-otp-24
rebar  3.16.1

And I also needed to brew install libxsl xmlto.

You can also do without a custom .config file and just tail a logfile, whose full path is printed just as RabbitMQ finishes booting up.)

Conclusion

One lesson of this investigation was that it’s often most effective to just dive into the source code itself. I was also reassured by how relatively approachable foreign codebases can be, when written in a functional language, Erlang & Elixir in this case.

While spelunking around RabbitMQ’s code I also realised how somewhat arbitrary the current implementation of Keycloak support is. I don’t think that the token format presented in issue #36, which then prompted the PR for Keycloak support, is in any way specific to Keycloak. Rather, the format shown in the issue, with "permissions" nested under an "authorization" field, is just a possible format that the issue-author just happened to be using, and can be easily replicated as shown in below image.

In short, I don’t think it’s warranted that RabbitMQ expects this exact structure to be present. Perhaps the existing "scope" field, also used by UAA, could be extended to also allow such nested structures.

And another lesson is, as always, to RTM.

Jupyter tips

Tue, 25 May 2021 00:00:00 GMT

As I do data-related projects only on a semi-regular basis, I often find myself regrouping from the context switch. There’s not much one can do about the inherent complexity of the subject, but maybe we can lessen the incidental one.

One of the things I often do is amend my Jupyter workflow with slight ergonomic improvements. Gathering the following tips will help my future self make this context switch a bit easier; maybe you’ll find them helpful as well.

Stable test-set split

One thing I often find myself in need of is a future-stable ID for a test-set split and I haven’t found an option in pandas.train_test_split that would offer this. Let me know if I missed it, or if there’s an equivalent in some other package. In lieu of that I often use the following split_train_test_by_id, which splits the data based on each record’s last-hash-byte membership in the [0, test_ratio*256) range.

from hashlib import md5

def _is_in_test_set(id_, test_ratio):
  id_hash = md5(id_.encode('utf-8')).digest()
  return id_hash[-1] < test_ratio * 256

def split_train_test_by_id(data, test_ratio, id_column):  in_test_set = data[id_column].apply(
    lambda id_: _is_in_test_set(id_, test_ratio))
  return data.loc[~in_test_set], data.loc[in_test_set]

df_train, df_test = split_train_test_by_id(df, test_ratio=0.2, id_column='an_id_column')

Shortcuts

One can rebind some built-in commands either A) through the web interface or B) via the "keys" object in ~/.jupyter/nbconfig/notebook.json. Any new, custom shortcuts should go to ~/.jupyter/custom/. Or, each such %%javascript cell should be execute per notebook.

%%javascript
Jupyter.keyboard_manager.edit_shortcuts.add_shortcut('ctrl-alt-k', {
  help: 'Kill line',
  help_index: 'zz',
  handler: function(env) {
    var cm = env.notebook.get_selected_cell().code_mirror
    cm.execCommand('goLineStart')
    cm.execCommand('killLine')
    cm.execCommand('delCharAfter')
    return false
  }}
)

As mentioned, rather than keep copy pasting above custom shortcuts between notebooks, a better solution would be to have them automatically set on startup.

$([IPython.events]).on('app_initialized.NotebookApp', function(){
  // WIP: have the command duplicate a line robustly.
  CodeMirror.keyMap.macDefault["cmd-shift-d"] = function(cm){
    var current_cursor = cm.doc.getCursor();
    var line_content = cm.doc.getLine(current_cursor.line);
    CodeMirror.commands.goLineEnd(cm);
    CodeMirror.commands.newlineAndIndent(cm);
    cm.doc.replaceSelection(line_content);
    cm.doc.setCursor(current_cursor.line + 1, current_cursor.ch);
  };
});

Better plotting defaults

Arguably better plotting defaults, especially the usage of ConciseDateConverter that keeps the axes from being overcrowded and unreadable.

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
plt.rcParams["figure.figsize"] = (16, 9)
sns.set()

import datetime
import matplotlib.dates as mdates
import matplotlib.units as munits
converter = mdates.ConciseDateConverter()munits.registry[np.datetime64] = converter
munits.registry[datetime.date] = converter
munits.registry[datetime.datetime] = converter

Bash defer idiom

Thu, 23 Apr 2020 00:00:00 GMT

One use case of trap in Bash is the following snippet:

tempfile=/tmp/tmpdata
function cleanup() {
    rm -f $tempfile
}
trap cleanup EXIT

The highlighted line in particular might be considered an equivalent of a defer statement in, say, Swift or Go.

awk's seen idiom

Mon, 13 May 2019 00:00:00 GMT

I wanted to extract distinct values of a particular column in a csv file, and just found awk has a concise idiom to do just that. Say you want to operate on the 2nd column, hence $2 in the following snippet. This is all it takes:

awk -F',' '!seen[$2]++' a.csv

seen is merely an example. You may use (at least) anything from the [A-Za-z_]+ domain.

Automated permissions testing

Wed, 06 Sep 2017 00:00:00 GMT

Previously we’ve covered deeplinks with automated tests. Let’s now do the same for permissions. We’ll only showcase doing so for notifications, but the same would apply to others: camera, photos, location, calendar, reminders, microphone, etc.

Prerequisites

First we need to empower UI testing with ability to interact with SpringBoard; and thus be able to delete the app-under-test between test cases.

Download private headers XCUIApplication.h and XCUIElement.h from WebDriverAgent. Then add a bridge header file, we’ll name it Bridge.h, that imports the downloaded headers:

// Bridge.h
#import "XCUIApplication.h"
#import "XCUIElement.h"

For the test-target, set Objective-C Bridging Header to ${ROOT}/<MyAppTests>/.../Bridge.h:

Now we start using the newly gotten private APIs:

import XCTest

final class Springboard {
  static let springboard = XCUIApplication(privateWithPath: nil, bundleID: "com.apple.springboard")!

Add a helper that deletes our app-under-test:

class func deleteMyApp() {
  XCUIApplication().terminate()

  // Resolve the query for the SpringBoard rather than launching it
  springboard.resolve()

  let icon = springboard.icons["MyApp"]
  XCTAssert(icon.isHittable)

  // Bring up the little "X" button to delete the app
  icon.press(forDuration: 1.3)

  // The little "X" button is, it seems, not exposed directly
  let xButtonCoordinate = CGVector(dx: (icon.frame.minX + 3) / springboard.frame.maxX,
                                   dy: (icon.frame.minY + 3) / springboard.frame.maxY)
  // Delete the app
  springboard.coordinate(withNormalizedOffset: xButtonCoordinate).tap()
  springboard.alerts.buttons["Delete"].tap()

  // Stop icons from jiggling
  XCUIDevice.shared().press(.home)
}

(Sidenote: why class rather than static? First one is 1 character shorter. Joking aside, it’s a toss-up since Springboard class is final.)

We also don’t have to hardcode the app name and can replace above "MyApp" with below myAppName:

private class var myAppName: String {
  // XCUIApplication().path example:
  // "/Users/{name}/Library/Developer/Xcode/DerivedData/MyApp-{someID}/Build/Products/Debug-iphonesimulator/MyApp.app"
  let appName = XCUIApplication().path
    .components(separatedBy: "/").last!
    .components(separatedBy: ".").dropLast()
  return Array(appName).first!
}

Finally, verify everything works thus far by calling Springboard.deleteMyApp() in, say, test class’s setUp(). Here’s how it should look like:

Permissions testing

We now have all prerequisites for permissions testing. With the private XCUIApplication(privateWithPath:bundleID:) constructor at our disposal, we call it with "com.apple.Preferences" as the bundleID:. It returns us a handle on Settings.app, and with it we get to reproducibly set our app’s notification settings and then verify that the app responds accordingly.

final class Springboard {
  static let springboard = XCUIApplication(privateWithPath: nil, bundleID: "com.apple.springboard")!
  static let settings = XCUIApplication(privateWithPath: nil, bundleID: "com.apple.Preferences")!  static let settingsIcon = springboard.icons["Settings"]

First add a helper function:

private class func openSettings() {
  // Make sure home screen is frontmost
  XCUIApplication().terminate()

  // Go to SpringBoard's first screen
  XCUIDevice.shared().press(.home)
  Thread.sleep(forTimeInterval: 0.5)
  XCTAssert(settingsIcon.isHittable)

  // Launch Settings.app from a known state
  settings.terminate()

  settingsIcon.tap()
}

Finally, notifications. These are slightly more tricky, as, besides notifications being ON or OFF, we also have to consider the 3rd state, when the app hasn’t yet requested notifications permissions and thus might not be listed in Settings. Taking this aside for now, here’s how to turn notifications ON or OFF:

enum NotificationsState {
  case on, off
}

class func turnNotifications(_ desiredState: NotificationsState) {
  openMyAppSectionInSettings()
  settingsTable.cells.staticTexts["Notifications"].tap()

  var currentState: NotificationsState {
    let soundsCell = settingsTable.cells.staticTexts["Sounds"]
    return soundsCell.exists && soundsCell.isHittable
      ? .on
      : .off
  }

  if (desiredState == .on && currentState == .off)
    || (desiredState == .off && currentState == .on) {
    let allowNotificationsSwitch = settings.switches.element(boundBy: 0)
    allowNotificationsSwitch.tap()
  }
}

See the sample project which puts the above together. Also note that not all permissions require deletion of the app; e.g. location permissions can be reset in Settings > General:

class func resetLocationAndPrivacy() {
  openSettings()

  settings.tables.staticTexts["General"].tap()
  settings.tables.staticTexts["Reset"].tap()
  settings.tables.staticTexts["Reset Location & Privacy"].tap()
  settings.buttons["Reset Warnings"].tap()
  settings.terminate()
}

But Wait, There’s More!

Not only Settings, we can interact with other Apple’s apps as well. Below are some of the bundle IDs.

App Name	| Bundle ID
---------------------------------
App Store	| com.apple.AppStore
Calculator	| com.apple.calculator
Calendar	| com.apple.mobilecal
Camera		| com.apple.camera
Clock		| com.apple.mobiletimer
Compass		| com.apple.compass
Contacts	| com.apple.MobileAddressBook
FaceTime	| com.apple.facetime
Files       | com.apple.DocumentManager # iOS 11+
iCloud Drive| com.apple.iclouddriveapp  # iOS 10
Find Friends| com.apple.mobileme.fmf1
Find iPhone	| com.apple.mobileme.fmip1
Health		| com.apple.health
Home		| com.apple.home
iBooks		| com.apple.ibooks
iTunes Store| com.apple.mobilestore
Mail		| com.apple.mobilemail
Maps		| com.apple.Maps
Messages	| com.apple.mobilesms
Music		| com.apple.Music
News		| com.apple.news
Notes		| com.apple.mobilenotes
Phone		| com.apple.mobilephone
Photos		| com.apple.mobileslideshow
Podcasts	| com.apple.podcasts
Reminders	| com.apple.reminders
Safari		| com.apple.mobilesafari
Settings	| com.apple.Preferences
Siri        | com.apple.SiriViewService
Stocks		| com.apple.stocks
Tips		| com.apple.tips
TV			| com.apple.tv
Videos		| com.apple.videos
Voice Memos	| com.apple.voicememos
Wallet		| com.apple.passbook
Watch		| com.apple.bridge
Weather		| com.apple.weather

Automated deeplinks testing

Mon, 28 Aug 2017 00:00:00 GMT

UI testing under iOS 11 has the ability to interact with other installed apps. Specifically, XCUIApplication got a new constructor that takes a bundle ID. Let’s see how we use this to put our deeplinks under automated tests.

Overview

What you basically do is create a helper-app, MyHelperApp in below examples, that merely holds UIButtons for all the deeplinks your main app, MyApp, should support. For MyApp you register a URL scheme, and for HelperApp you whitelist that very same scheme under the LSApplicationQueriesSchemes key in Info.plist:

The new XCUIApplication constructor allows you to open MyHelperApp…

let helperApp = XCUIApplication(bundleIdentifier: "org.merela.MyHelperApp")
helperApp.activate()
helperApp.buttons[deeplink].tap()

…and there open a particular deeplink that MyApp supports:

let url = URL(string: "\(myAppBundleId)://\(buttonTitle)")
UIApplication.shared.open(url)

Finally, assert that the deeplink opened MyApp in the expected state.

With Xcode 9, check out the sample project. To try it out, first install the HelperApp and then run tests for the MyApp target. Also make sure Simulator’s ‘Hardware Keyboard’ is disabled.

CI considerations

Xcode 9 GM will be released soon, but until then you have to switch to Xcode-beta’s toolchain, to be used by xcodebuild commands further below:
sudo xcode-select -s /Applications/Xcode-beta.app/Contents/Developer

This proof-of-concept script will make sure helper-app is always installed and ready to be used by main-app’s UI tests:

HELPER_APP='MyHelperApp'
MAIN_APP='MyApp'

BUILD_DIR="/tmp/$HELPER_APP"
DESTINATION="platform=iOS Simulator,name=iPhone 7 Plus,OS=11.0"

# Build an .app for the helper-app
xcodebuild build                       \
           -destination "$DESTINATION" \
           -target $HELPER_APP         \
           -scheme $HELPER_APP         \
           CONFIGURATION_BUILD_DIR=$BUILD_DIR

# Install the helper-app onto the Simulator
xcrun simctl install booted "$BUILD_DIR/$HELPER_APP.app"

# UI test the main-app
xcodebuild test -scheme $MAIN_APP -destination "$DESTINATION"

Conclusion

One other new API worth mentioning is XCTContext.runActivity(named:block:), which logically groups subparts of each test-case, makes these groups visible in logs, and thus eases debugging:

There’s plenty more: performance improvements (firstMatch), screenshots, attachments, etc. Check out WWDC 2017’s What’s New in Testing.

Distilled iOS development tips

Tue, 20 Jun 2017 00:00:00 GMT

How to get more accurate Xcode test coverage

A next step from testing your code is knowing how much of your code is covered by these tests. To do that, you have to first open the Edit Scheme and tell Xcode to Gather coverage data. (This setting is as of Xcode 8.2.1 disabled by default.)

Running unit tests as is would give you inaccurate coverage, as it would take into account AppDelegate and everything it touches. (Because app process hosts these unit tests and they are executed after receiving notification for UIApplicationDidFinishLaunching.) This is easily seen on a new project, using the Single View Application template that comes with AppDelegate.swift and ViewController.swift files. As you see below, I’ve also added AnIntegration.swift that contains only one empty function which gets called from AppDelegate and so, by definition, has 100% coverage:

To get accurate code coverage first add a AppDelegate barebones replica to your test-target:

import UIKit

class TestAppDelegate: UIResponder, UIApplicationDelegate {
  var window: UIWindow?
  func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool {
    return true
  }
}

Then add main.swift to the app-target in and finally tell Xcode to, when testing, replace AppDelegate with the above TestAppDelegate:

import Foundation
import UIKit

let argc = CommandLine.argc
let argv = UnsafeMutableRawPointer(CommandLine.unsafeArgv).bindMemory(to: UnsafeMutablePointer<Int8>.self, capacity: Int(argc))
let appDelegateClass: AnyClass = ProcessInfo.isUnitTesting
  ? NSClassFromString("MyAppTests.TestAppDelegate")!
  : AppDelegate.self

UIApplicationMain(argc, argv, nil, NSStringFromClass(appDelegateClass))

The ProcessInfo.isUnitTesting above is an extension on ProcessInfo:

import Foundation

extension ProcessInfo {
  static var isUnitTesting: Bool {
    return processInfo.environment["XCTestConfigurationFilePath"] != nil
  }
}

Since you supplied main.swift file, you now also have to remove the @UIApplicationMain attribute that’s right above your AppDelegate:

@UIApplicationMain // remove thisclass AppDelegate: UIResponder, UIApplicationDelegate {

Finally, run the tests again and coverage of AnIntegration is now at the actual value of 0%:

How to make Xcode downloads robuster

This comes handy especially when on an unreliable network. Sign in to AppStoreConnect, open Chrome DevTools, and click the download link. The last request you’ll see in the DevTools’ Network tab will be the download request you just made. Right click on it, choose ‘Save as cURL’, and wrap the copied request in a loop:

for ((i=1;i<=100;i++)); do
curl 'https://download.developer.apple.com/Developer_Tools/Xcode_12_beta_3/Xcode_12_beta_3.xip' \
  -H 'authority: download.developer.apple.com' \
  -H 'pragma: no-cache' \
  -H 'cache-control: no-cache' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: {your-agent}' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'sec-fetch-site: same-site' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-user: ?1' \
  -H 'sec-fetch-dest: document' \
  -H 'referer: https://developer.apple.com/download/' \
  -H 'accept-language: en-US,en;q=0.9,sl;q=0.8,de;q=0.7' \
  -H 'cookie: {your-cookie}' \
  --compressed \
  -o Xcode-beta.xip \
  -L \
  --continue-at -sleep 1
done

The {your-cookie} part should be filled in for you. The cookie can be also be fetched through other means, from Chrome DevTools or a Chrome extension. The important part above is the highlighted --continue-at flag. It instructs curl to remember the offset of already-transferred bytes and, if needed, resume the transfer of the same file from this offset.

How to streamline Xcode upgrades

It seems unavoidable for new Xcode versions to override any custom keybindings. Ditto for a custom, brighter text-cursor, which is pretty much a necessity when using a darker background. And let’s not forget the plugins. Since Xcode updates aren’t that rare, here is a script of all the steps:

update_xcode {
  # Change if not using default naming, "Xcode" and "Xcode-beta"
  _update_xcode "Xcode"
  [ -d "/Applications/Xcode-beta.app/" ] && _update_xcode "Xcode-beta"
}

_update_xcode {
  XCODE_PATH="/Applications/$1.app"
  PLUGINS_PATH="$HOME/Library/Application Support/Developer/Shared/Xcode/Plug-ins"
  SOURCE_PATH="$HOME/path/to/your/keybindings_etc"

  # Update plugin UUIDs
  uuid=$(defaults read $XCODE_PATH/Contents/Info DVTPlugInCompatibilityUUID)
  find "$PLUGINS_PATH" -name Info.plist -maxdepth 3 | xargs -I{} defaults write {} DVTPlugInCompatibilityUUIDs -array-add "$uuid"
  # Restore keybindings
  sudo cp -f $SOURCE_PATH/IDETextKeyBindingSet.plist $XCODE_PATH/Contents/Frameworks/IDEKit.framework/Versions/A/Resources
  # Restore white text-cursor
  sudo cp -f $SOURCE_PATH/DVTIbeamCursor.tiff $XCODE_PATH/Contents/SharedFrameworks/DVTKit.framework/Versions/A/Resources/

  restart "$1"
}

restart {
  osascript -e 'quit app "$1"'
  open -a "$1"
}

Xcode 8+ update

Apple hardened Xcode 8 and later against XcodeGhost and thus broke many plugins. If you value your plugins over safety, then you’ll have to re-sign Xcode:

sudo codesign -f -s "Some certificate e.g. 'iPhone Developer: Your Name (AAAAAAAA)'" /Applications/Xcode.app

Not ideal, but until Xcode supports more than mere text-extensions, I think the tradeoff is worth it.

How to use a git hook to avoid pushing Xcode’s mess onto others

A good boy scout “leaves the campground cleaner than he found it”. Or at least not leave the campground messier than he found it. In iOS this includes those ‘misplaced views’ warnings that Xcode sometimes unhelpfully causes when you merely peek at .xib files.

Future Xcode releases will hopefully fix this. But for now it’s still up to us to fix it. The following Git hook ensures all team members keep the campground clean for others:

#!/usr/bin/env bash
set -e

failed=0

misplaced_pattern='misplaced="YES"'
if git diff-index -p -M --cached HEAD -- '*.xib' '*.storyboard' | grep '^+' | egrep "$misplaced_pattern" >/dev/null 2>&1
then
  echo "PUSH REJECTED, you have misplaced views:" >&2
  echo "$(git --no-pager grep -l --cached "$misplaced_pattern" '*.xib' '*.storyboard')" >&2
  echo ""
  failed=1
fi

# Potentially more checks that raise the 'failed' flag.

exit $failed

How to scrape iOS URL schemas

Say you have to find out URL schemas of 3rd party iOS apps, many of them. First download these apps using the macOS iTunes. Right-click one of the downloaded apps and select Show in Finder.

This will open the folder of .ipa files at ~/Music/iTunes/iTunes Media/Mobile Applications which you then copy to another folder of your choosing. In this folder you now run the following script:

# Usage: $0 <directory of .ipa files> <output file with (app-name URL-scheme) pairs>
for file in ./*; do
  dir="$(basename $file .ipa)";
  unzip -uqd "$dir" $file;
  app_name_path="$(find "$dir/Payload" -maxdepth 1 -name '*.app')"
  app_name="$(basename "$app_name_path")"
  plist=$(plutil -extract CFBundleURLTypes.0.CFBundleURLSchemes.0 xml1 -o - "$app_name_path/Info.plist")
  scheme=$(echo "$plist" | grep \<string\>)
  echo "$app_name $scheme" >> ../schemas
done

How to lessen compiler’s shortcoming

One weird Swift trick to not have to come up with silly and inconsistent names for the unwrapped weak self:

iMightOutliveSelf { [weak self] in
  guard let `self` = self else { return }
  self.doSomething()
}

Fellow developers might hate you for doing this, for it is a Swift compiler bug. Swift does allow backticks to use a otherwise reserved word in other cases though.

Update: This has been fixed in Swift 4.2 and now you can unwrap self as you would any other variable, without backticks:

iMightOutliveSelf { [weak self] in
  guard let self = self else { return }  self.doSomething()
}

Pre-recorded test data

There’s a Test Application Data setting for providing test data with which you can launch tests from a known state. Among other uses, having such known state is pretty much essential for any thorough testing of Core Data migrations.

First go to Window > Devices and download the data container from your device:

Then open the Edit Scheme... screen and load the just download .xcappdata:

Note that from Xcode 6 to including Xcode 7.3 this might not be working on the iOS Simulator.