ruby


A Home-Based Ubuntu 16.04 Production Server with Salvaged Equipment

Preface

As I’ve often griped before, my cloud service provider (cloudatcost.com) is not exactly reliable. I’m currently on Day 3 waiting for their tech support to address several downed servers. Three days isn’t even that bad considering I’ve waited up to two weeks in the past. In any case, though I wish them success, I’m sick of their nonsense and am starting to migrate my servers out of the cloud and into my house. As a cloud company that routinely loses its customers’ data, it’s prudent to prepare for their likely bankruptcy and closure.

I have an old smashed-up AMD Quad Core laptop I’m going to use as a server. The screen was totally broken, so as a laptop it’s kind of useless anyway. It’s a little lightweight on resources (only 4 GB of RAM), but this is much more than I’m used to. I used unetbootin to create an Ubuntu 16.04 bootable USB and installed the base system.

What follows are the common steps I take when setting up a production server. This bare minimum approach is a process I repeat frequently, so it’s worth documenting here. Once the OS is installed…

Change the root password

The install should put the created user into the sudo group. Change the root password with that user:

1
2
3
sudo su
passwd
exit

Update OS

An interesting thing happened during install… I couldn’t install additional software (i.e., open-ssh), so I skipped it. When it came time to install vim, I discovered I didn’t have access to any of the repositories. The answer here shed some light on the situation, but didn’t really resolve anything.

I ended up copying the example sources.list from the documentation to fix the problem:

1
sudo cp /usr/share/doc/apt/examples/sources.list /etc/apt/sources.list

I found out later that this contained all repositories for Ubuntu 14.04. So I ended up manually pasting this in /etc/apt/sources.list:

1
2
3
4
5
6
7
8
9
10
11
12
# deb cdrom:[Ubuntu 16.04 LTS _Xenial Xerus_ - Release amd64 (20160420.1)]/ xenial main restricted
deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
# deb http://archive.ubuntu.com/ubuntu xenial-proposed main restricted universe multiverse
deb http://archive.canonical.com/ubuntu xenial partner
deb-src http://archive.canonical.com/ubuntu xenial partner

After that, update/upgrade worked (make sure it doesn’t actually work before messing around):

1
2
sudo apt update
sudo apt upgrade

Install open-ssh

The first thing I do is configure my machine for remote access. As above, I couldn’t install open-ssh during the OS installation, for some reason. After sources.list was sorted out, it all worked:

1
sudo apt-get install openssh-server

Check the official Ubuntu docs for configuration tips.

While I’m here, though, I need to set a static IP…

1
sudo vi /etc/network/interfaces

Paste this (or similar) under # The primary network interface, as per lewis4u.

1
2
3
4
5
auto enp0s25
iface enp0s25 inet static
address 192.168.0.150
netmask 255.255.255.0
gateway 192.168.0.1

Then flush, restart, and verify that the settings are correct:

1
2
3
sudo ip addr flush enp0s25
sudo systemctl restart networking.service
ip add

Start ssh

My ssh didn’t start running automatically after install. I did this to make ssh run on startup:

1
sudo systemctl enable ssh

And then I did this, which actually starts the service:

1
sudo service ssh start

Open a port on the router

This step, of course, depends entirely on the make and model of router behind which the server is operating. For me, I access the adminstrative control panel by logging in at 192.168.0.1 on my LAN.

I found the settings I needed to configure on my Belkin router under Firewall -> Virtual Servers. I want to serve up web apps (both HTTP and HTTPS) and allow SSH access. As such, I configured three access points by providing the following information for each:

  1. Description
  2. Inbound ports (i.e., 22, 80, and 443)
  3. TCP traffic type (no UDP)
  4. The private/static address I just set on my server
  5. Inbound private ports (22, 80, and 443 respectively)

Set up DNS

Again, this depends on where you registered your domain. I pointed a domain I have registered with GoDaddy to my modems’s IP address, which now receives requests and forwards them to my server.

Login via SSH

With my server, router, and DNS all properly configured, I don’t need to be physically sitting in front of my machine anymore. As such, I complete the following steps logged in remotely.

Set up app user

I like to have one user account control app deployment. Toward that end, I create an app user and add him to the sudo group:

1
2
sudo adduser app
sudo adduser app sudo

Install the essentials

git

Won’t get far without git:

1
sudo apt install git

vim

My favourite editor is vim, which is not installed by default.

1
sudo apt install vim

NERDTree

My favourite vim plugin:

1
2
3
4
mkdir -p ~/.vim/autoload ~/.vim/bundle
cd ~/.vim/autoload
wget https://raw.github.com/tpope/vim-pathogen/HEAD/autoload/pathogen.vim
vim ~/.vimrc

Add this to the .vimrc file:

1
2
3
4
call pathogen#infect()
map <C-n> :NERDTreeToggle<CR>
set softtabstop=2
set expandtab

Save and exit:

1
2
cd ~/.vim/bundle 
git clone https://github.com/scrooloose/nerdtree.git

Now, when running vim, hit ctrl-n to toggle the file tree view.

Docker

Current installation instructions can be found here. The distilled process is as follows:

1
2
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

You can verify the key fingerprint:

1
sudo apt-key fingerprint 0EBFCD88

Which should return something like this:

1
2
3
4
pub   4096R/0EBFCD88 2017-02-22
      Key fingerprint = 9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
      uid                  Docker Release (CE deb) <docker@docker.com>
      sub   4096R/F273FCD8 2017-02-22

Add repository and update:

1
2
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update

Install:

1
sudo apt install docker-ce

Create a docker user group:

1
sudo groupadd docker

Add yourself to the group:

1
sudo usermod -aG docker $USER

Add the app user to the group as well:

1
sudo usermod -aG docker $USER

Logout, login, and test docker without sudo:

1
docker run hello-world

If everything works, you should see the usual Hello, World! message.

Configure docker to start on boot:

1
sudo systemctl enable docker

docker-compose

This downloads the current stable version. Cross reference it with that offered here.

1
2
3
su
curl -L https://github.com/docker/compose/releases/download/1.14.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Install command completion while still root:

1
2
curl -L https://raw.githubusercontent.com/docker/compose/master/contrib/completion/bash/docker-compose -o /etc/bash_completion.d/docker-compose
exit

Test:

1
docker-compose --version

node

These steps are distilled from here.

1
2
3
cd ~
curl -sL https://deb.nodesource.com/setup_6.x -o nodesource_setup.sh
sudo bash nodesource_setup.sh

Now install:

1
sudo apt-get install nodejs build-essential

Ruby

The steps followed conclude with installing rails. I only install ruby:

1
sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev nodejs

Using rbenv:

1
2
3
4
5
6
7
8
9
10
11
cd
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
exec $SHELL

git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc
exec $SHELL

rbenv install 2.4.0

That last step can take a while…

1
2
rbenv global 2.4.0
ruby -v

Install Bundler:

1
gem install bundler

Done!

There it is… all my usual favourites on some busted up piece of junk laptop. I expect it to be exactly 1000% more reliable than cloudatcost.com.


Investigating Final Draft's XML document format with Ruby

So apparently there is no open source screenplay format. I was poking around and the closest I came was this and this, which ultimately led me to this. At the time of writing, the Open Screenplay Format (OSF) apparently doesn’t exist anymore.

The paranoid-conspiratorial side of me suspects that the nefarious folk at Final Draft are behind the OSF’s disappearance. In retaliation for their imagined meddling in the affairs of their competitors, I decided to straight-up jack their FDX (Final Draft-flavoured XML) file format and make it better.

First, I obtained a script.

Since I’d already been poking around a bit, I knew about fountain.io. They’ve got some sort of Markdown-flavoured screenplay-writing utility (which is awesome). It just so happens that they have a copy of Big Fish in FDX format. Perfect.

This is kind of what FDX looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<FinalDraft DocumentType="Script" Template="No" Version="1">
  <Content>
    <Paragraph Type="General">
      <Text>This is a Southern story, full of lies and fabrications, but truer for their inclusion.</Text>
    </Paragraph>
    <Paragraph Type="Transition">
      <Text>FADE IN</Text>
    </Paragraph>
    <Paragraph Type="Scene Heading">
      <Text>A RIVER.</Text>
    </Paragraph>
    <Paragraph Type="Action">
      <Text>We’re underwater, watching a fat catfish swim along.</Text>
    </Paragraph><Paragraph Type="Action">
    <Text>This is The Beast.</Text>
  </Paragraph>
  <Paragraph Type="Character">
    <Text>EDWARD (V.O.)</Text>
  </Paragraph>
  <Paragraph Type="Dialogue">
    <Text>There are some fish that cannot be caught. It’s not that they’re faster or stronger than other fish. They’re just touched by something extra. Call it luck. Call it grace. One such fish was The Beast.</Text>
  </Paragraph>
  <!-- And so forth... -->

Upon a cursory inspection, I quickly concluded that FDX is primarily concerned with the visual format of the exported screenplay (obviously). I, on the other hand, am only concerned with visual format insofar as it provides me clues as to how to import typeset screenplays and shoehorn them into a new format… something less XML and more JSON, perhaps.

Though I generally think XML is a pain to work with, it is well-suited for the kind of typesetting and document structuring that is the purview of Final Draft. Less so for a DevOps-driven movie studio that wants the ever-changing script to automatically orchestrate its own production. I’m not sure what the final Open Screenplay Format (I’m stealing the name now) will look like at this early stage of the game, but I do know that FDX will provide a good starting point.

As such, I need some basic information about FDX. I.e.,

  • The elements of which it is comprised,
  • The attributes of each of those elements,
  • And valid values for each of those attributes

It didn’t take long to realize that reading the FDX screenplay and cataloging this information by hand is dumb, so I whipped up this groovy little ad hoc Ruby script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
require 'nokogiri'
require 'pp'

f = File.open('Big Fish.fdx')
doc = Nokogiri::XML(f)
f.close

elems = doc.xpath("//*")

schema = {}
elems.each do |e|

  # Add a new element, if necessary
  schema[e.name] = {} unless schema.has_key? e.name

  # Get an element's attributes
  e.attributes.keys.each do |a|
    schema[e.name][e.attributes[a].name] = [] unless schema[e.name].has_key? e.attributes[a].name

    # Get valid attribute values
    schema[e.name][e.attributes[a].name] << e.attributes[a].value unless schema[e.name][e.attributes[a].name].include? e.attributes[a].value
  end
end

pp schema

All that produced this (simplified):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
{"FinalDraft"=>
  {"DocumentType"=>["Script"], "Template"=>["No"], "Version"=>["1"]},
  "Content"=>{},
  "Paragraph"=>
    {"Type"=>
      ["General",
       "Transition",
       "Scene Heading",
       "Action",
       "Character",
       "Dialogue",
       "Parenthetical"],
    "Alignment"=>["Center", "Right"],
    "FirstIndent"=>["0.00"],
    "Leading"=>["Regular"],
    "LeftIndent"=>["1.25"],
    "RightIndent"=>["-1.25"],
    "SpaceBefore"=>["0"],
    "Spacing"=>["1"],
    "StartsNewPage"=>["No"]},
  "Text"=>
    {"AdornmentStyle"=>["0"],
     "Background"=>["#FFFFFFFFFFFF"],
     "Color"=>["#000000000000"],
     "Font"=>["Courier Final Draft"],
     "RevisionID"=>["0"],
     "Size"=>["12"],
     "Style"=>[""]},
  "TitlePage"=>{},
  "HeaderAndFooter"=>
    {"FooterFirstPage"=>["No"],
     "FooterVisible"=>["No"],
     "HeaderFirstPage"=>["No"],
     "HeaderVisible"=>["Yes"],
     "StartingPage"=>["1"]},
  "Header"=>{},
  "DynamicLabel"=>{"Type"=>["Page #"]},
  "Footer"=>{},
  "PageLayout"=>
    {"BackgroundColor"=>["#FFFFFFFFFFFF"],
     "BottomMargin"=>["72"],
     "BreakDialogueAndActionAtSentences"=>["Yes"],
     "DocumentLeading"=>["Normal"],
     "FooterMargin"=>["36"],
     "ForegroundColor"=>["#000000000000"],
     "HeaderMargin"=>["36"],
     "InvisiblesColor"=>["#A0A0A0A0A0A0"],
     "TopMargin"=>["72"],
     "UsesSmartQuotes"=>["No"]},
  "AutoCastList"=>
    {"AddParentheses"=>["Yes"],
     "AutomaticallyGenerate"=>["No"],
     "CastListElement"=>["Cast List"]},
  "ElementSettings"=>
    {"Type"=>
      ["General",
       "Scene Heading",
       "Action",
       "Character",
       "Parenthetical",
       "Dialogue",
       "Transition",
       "Shot",
       "Cast List",
       "New Act"]},
  "FontSpec"=>
    {"AdornmentStyle"=>["0"],
     "Background"=>["#FFFFFFFFFFFF"],
     "Color"=>["#000000000000"],
     "Font"=>["Courier Final Draft"],
     "RevisionID"=>["0"],
     "Size"=>["12"],
     "Style"=>["", "AllCaps", "Underline+AllCaps"]},
  "ParagraphSpec"=>
    {"Alignment"=>["Left", "Right", "Center"],
     "FirstIndent"=>["0.00", "-0.10"],
     "Leading"=>["Regular"],
     "LeftIndent"=>["1.50", "3.50", "3.00", "2.50", "5.50"],
     "RightIndent"=>["7.50", "7.25", "5.50", "6.00", "7.10"],
     "SpaceBefore"=>["0", "24", "12", "120"],
     "Spacing"=>["1"],
     "StartsNewPage"=>["No", "Yes"]},
  "Behavior"=>
    {"PaginateAs"=>
      ["General",
       "Scene Heading",
       "Action",
       "Character",
       "Parenthetical",
       "Dialogue",
       "Transition"],
     "ReturnKey"=>["General", "Action", "Dialogue", "Scene Heading"],
     "Shortcut"=>["0", "1", "2", "3", "4", "5", "6", "7", "8", ""]}}

This enabled me to identify the most important script elements, which, upon inspection, are anything tagged with Paragraph and constrained by the Type attribute. That is, a movie (Big Fish at the very least) is comprised of the following:

  • General
  • Transition
  • Scene Heading
  • Action
  • Character
  • Dialogue
  • Parenthetical

These are the components of a screenplay that direct the action on screen. The rest appear mostly concerned with typesetting the document exported from Final Draft.

From here I will investigate the best way to structure the new and improved OSF. Stay tuned…