So apparently there is no open source screenplay format. I was poking around and the closest I came was this and this, which ultimately led me to this. At the time of writing, the Open Screenplay Format (OSF) apparently doesn’t exist anymore.

The paranoid-conspiratorial side of me suspects that the nefarious folk at Final Draft are behind the OSF’s disappearance. In retaliation for their imagined meddling in the affairs of their competitors, I decided to straight-up jack their FDX (Final Draft-flavoured XML) file format and make it better.

First, I obtained a script.

Since I’d already been poking around a bit, I knew about fountain.io. They’ve got some sort of Markdown-flavoured screenplay-writing utility (which is awesome). It just so happens that they have a copy of Big Fish in FDX format. Perfect.

This is kind of what FDX looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<FinalDraft DocumentType="Script" Template="No" Version="1">
  <Content>
    <Paragraph Type="General">
      <Text>This is a Southern story, full of lies and fabrications, but truer for their inclusion.</Text>
    </Paragraph>
    <Paragraph Type="Transition">
      <Text>FADE IN</Text>
    </Paragraph>
    <Paragraph Type="Scene Heading">
      <Text>A RIVER.</Text>
    </Paragraph>
    <Paragraph Type="Action">
      <Text>We’re underwater, watching a fat catfish swim along.</Text>
    </Paragraph><Paragraph Type="Action">
    <Text>This is The Beast.</Text>
  </Paragraph>
  <Paragraph Type="Character">
    <Text>EDWARD (V.O.)</Text>
  </Paragraph>
  <Paragraph Type="Dialogue">
    <Text>There are some fish that cannot be caught. It’s not that they’re faster or stronger than other fish. They’re just touched by something extra. Call it luck. Call it grace. One such fish was The Beast.</Text>
  </Paragraph>
  <!-- And so forth... -->

Upon a cursory inspection, I quickly concluded that FDX is primarily concerned with the visual format of the exported screenplay (obviously). I, on the other hand, am only concerned with visual format insofar as it provides me clues as to how to import typeset screenplays and shoehorn them into a new format… something less XML and more JSON, perhaps.

Though I generally think XML is a pain to work with, it is well-suited for the kind of typesetting and document structuring that is the purview of Final Draft. Less so for a DevOps-driven movie studio that wants the ever-changing script to automatically orchestrate its own production. I’m not sure what the final Open Screenplay Format (I’m stealing the name now) will look like at this early stage of the game, but I do know that FDX will provide a good starting point.

As such, I need some basic information about FDX. I.e.,

• The elements of which it is comprised,
• The attributes of each of those elements,
• And valid values for each of those attributes

It didn’t take long to realize that reading the FDX screenplay and cataloging this information by hand is dumb, so I whipped up this groovy little ad hoc Ruby script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

require 'nokogiri'
require 'pp'

f = File.open('Big Fish.fdx')
doc = Nokogiri::XML(f)
f.close

elems = doc.xpath("//*")

schema = {}
elems.each do |e|

  # Add a new element, if necessary
  schema[e.name] = {} unless schema.has_key? e.name

  # Get an element's attributes
  e.attributes.keys.each do |a|
    schema[e.name][e.attributes[a].name] = [] unless schema[e.name].has_key? e.attributes[a].name

    # Get valid attribute values
    schema[e.name][e.attributes[a].name] << e.attributes[a].value unless schema[e.name][e.attributes[a].name].include? e.attributes[a].value
  end
end

pp schema

All that produced this (simplified):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

{"FinalDraft"=>
  {"DocumentType"=>["Script"], "Template"=>["No"], "Version"=>["1"]},
  "Content"=>{},
  "Paragraph"=>
    {"Type"=>
      ["General",
       "Transition",
       "Scene Heading",
       "Action",
       "Character",
       "Dialogue",
       "Parenthetical"],
    "Alignment"=>["Center", "Right"],
    "FirstIndent"=>["0.00"],
    "Leading"=>["Regular"],
    "LeftIndent"=>["1.25"],
    "RightIndent"=>["-1.25"],
    "SpaceBefore"=>["0"],
    "Spacing"=>["1"],
    "StartsNewPage"=>["No"]},
  "Text"=>
    {"AdornmentStyle"=>["0"],
     "Background"=>["#FFFFFFFFFFFF"],
     "Color"=>["#000000000000"],
     "Font"=>["Courier Final Draft"],
     "RevisionID"=>["0"],
     "Size"=>["12"],
     "Style"=>[""]},
  "TitlePage"=>{},
  "HeaderAndFooter"=>
    {"FooterFirstPage"=>["No"],
     "FooterVisible"=>["No"],
     "HeaderFirstPage"=>["No"],
     "HeaderVisible"=>["Yes"],
     "StartingPage"=>["1"]},
  "Header"=>{},
  "DynamicLabel"=>{"Type"=>["Page #"]},
  "Footer"=>{},
  "PageLayout"=>
    {"BackgroundColor"=>["#FFFFFFFFFFFF"],
     "BottomMargin"=>["72"],
     "BreakDialogueAndActionAtSentences"=>["Yes"],
     "DocumentLeading"=>["Normal"],
     "FooterMargin"=>["36"],
     "ForegroundColor"=>["#000000000000"],
     "HeaderMargin"=>["36"],
     "InvisiblesColor"=>["#A0A0A0A0A0A0"],
     "TopMargin"=>["72"],
     "UsesSmartQuotes"=>["No"]},
  "AutoCastList"=>
    {"AddParentheses"=>["Yes"],
     "AutomaticallyGenerate"=>["No"],
     "CastListElement"=>["Cast List"]},
  "ElementSettings"=>
    {"Type"=>
      ["General",
       "Scene Heading",
       "Action",
       "Character",
       "Parenthetical",
       "Dialogue",
       "Transition",
       "Shot",
       "Cast List",
       "New Act"]},
  "FontSpec"=>
    {"AdornmentStyle"=>["0"],
     "Background"=>["#FFFFFFFFFFFF"],
     "Color"=>["#000000000000"],
     "Font"=>["Courier Final Draft"],
     "RevisionID"=>["0"],
     "Size"=>["12"],
     "Style"=>["", "AllCaps", "Underline+AllCaps"]},
  "ParagraphSpec"=>
    {"Alignment"=>["Left", "Right", "Center"],
     "FirstIndent"=>["0.00", "-0.10"],
     "Leading"=>["Regular"],
     "LeftIndent"=>["1.50", "3.50", "3.00", "2.50", "5.50"],
     "RightIndent"=>["7.50", "7.25", "5.50", "6.00", "7.10"],
     "SpaceBefore"=>["0", "24", "12", "120"],
     "Spacing"=>["1"],
     "StartsNewPage"=>["No", "Yes"]},
  "Behavior"=>
    {"PaginateAs"=>
      ["General",
       "Scene Heading",
       "Action",
       "Character",
       "Parenthetical",
       "Dialogue",
       "Transition"],
     "ReturnKey"=>["General", "Action", "Dialogue", "Scene Heading"],
     "Shortcut"=>["0", "1", "2", "3", "4", "5", "6", "7", "8", ""]}}

This enabled me to identify the most important script elements, which, upon inspection, are anything tagged with Paragraph and constrained by the Type attribute. That is, a movie (Big Fish at the very least) is comprised of the following:

• General
• Transition
• Scene Heading
• Action
• Character
• Dialogue
• Parenthetical

These are the components of a screenplay that direct the action on screen. The rest appear mostly concerned with typesetting the document exported from Final Draft.

From here I will investigate the best way to structure the new and improved OSF. Stay tuned…

Preamble

My buddy’s brother is the executive chef at a high-end restaurant near where I live. He’s passionate about his work and his industry. His role has enabled him to experiment in innovative ways. For example, he recently turned an old refrigerator into a smoker. He smokes salmon, bacon, and whatever he feels like. One day I’m sure he’ll move on to bigger and better things. When he does, he’ll take his smoker, his menus, recipes, and whatever else he created in the time he was employed. If he were a programmer, he’d be straight up robbed of all that stuff.

Programmers get screwed all the time. We typically agree to work for a salary and not compete in an industry for two years (or whatever). When we leave a position, we have nothing to show for our time, because we can’t take our work with us. When we go on to a new position, we find ourselves recreating some variation of the same old boring wheel that our previous employer had us build. That’s stupid, and it doesn’t help anyone, employer or programmer.

Programmers are writers. Writers get royalties, but programmers don’t. Don’t sweat it though, we’ve got something better: open source software. That follows us wherever we go. No one owns it. No one can take it from you. You can deploy it anywhere and not get robbed by old-fashioned, short-sighted employers who think software is a CD ROM sold in a box.

If you’re the keyboard-banging-Shakespeare-monkey-type programmer, nothing past this point is going to make any sense. If you’re the type of programmer who creates worlds with words and uses computer language as a medium of self expression, don’t be a sucker - develop open source until your reputation allows you to impose your own licence agreements. Sign up for GitHub and get started now.

In 2013 I was awarded a very nice grant to conduct research into semantic search. As part of my obligations to the funding agency, I was required to file two progress reports over the two year funding period. What follows is the final report. I post it for anyone who’s interested, though it’s especially relevant to programmers. I may have got screwed a little, but at the end of it all I kind of feel like Jacob in his dealings with his uncle Laban.

Behold the power of open source!

Introduction

It is with some consternation that I submit this document, my final report, covering my research into semantic search under the auspices of HireGround Software Solutions. Sadly, this research was cut short after being illegally terminated from my position prior to my return from provincially-protected parental leave. As such, the bulk of this report will review the activities that took place during my tenure, but prior to my termination. There is also discussion of the work I conducted independently while on leave and the avenue I would have pursued had I not been forced into this unjust and senseless situation.

Motivation

HireGround develops and supports human resource software designed to expedite candidate selection. The acquisition and processing of candidate resumes can be an expensive and resource-intensive challenge for organizations, especially when received in large volumes. Positions for which there is a large talent pool from which to draw can attract the interest of hundreds, or even thousands of candidates. Determining the best people to fill these positions is an arduous task. Even when the number of applicants is relatively small (say less than 20), hiring managers are tasked with sifting through data that is inconsistent in format and quality. The nature of the material that drive their decisions - resumes written in natural language - provides a fruitful environment in which to apply semantic search techniques.

Year One

Two challenges were addressed in the first year of funding:

1. Applicant matching: processing large volumes of resumes in an efficient and consistent manner so that the most qualified applicants, as measured against a given job description, were immediately identified.
2. Named Entity Recognition: analyzing resumes so that a candidate’s personal information and qualifications could be identified and structured. Had my research been allowed to continue, this would have provided the context with which existing semantic search techniques could be improved and adapted for new purposes.

Applicant Matching

The applicant matching technique I developed was my first contribution to HireGround’s flagship product: the StartDate Application Tracking System (ATS). StartDate provides the means with which hiring managers track and process job applicants. As previously stated, some positions attract a lot of interest. My applicant matching technique simply compares incoming resumes to the job description provided. The resumes that most closely match that description are the ones that float to the top of the list. This technique is akin to what hiring managers do manually: identify the key requirements of the role to be filled and match them to the skills stated on an applicant’s resume. The closer the match, the more likely an applicant is qualified to fill the role.

Here, manual processing is mechanized. The need for a manual, human-driven search is eliminated in favour of speed, consistency, and economy.

Named Entity Recognition

Though effective, the candidate matching technique applied above is crude. In partnership with my co-worker, Dr. Yanfen Hao (NSERC), we developed the RESume Named Entity Recognizer (Resner). Dr. Hao implemented a language model tailor-made for extracting applicant information from resumes. I was tasked with preparing the product for market and ensuring the extracted data was structured in accordance with the HR Open Standard.

As with human understanding of a given text, the machine’s treatment of natural language must be decided within context. Resumes are an interesting literary genre (of sorts), because each of the sections therein provide a unique context that demands its own understanding. Prior to parental leave, Resner had been integrated into the StartDate product and was hosted at resner.ca. Sadly, at the time of writing, Resner is no longer available at that domain.

Year Two

The first four months prior to my parental leave were spent focused on Resner. One month of which was spent preparing HireGround’s developers to maintain the software while I was on leave. My intention, upon returning to my position, was to implement a novel search technique designed to exploit the context Resner provides. Conceptually, the implementation was worked out, but never formalized or proven in software while present at HireGround (more on this later).

Parental Leave

I include a summary of my activities while on leave to underscore my dismay at the senselessness of my dismissal. Though not exclusive to the field of human resources, I was able to develop some basic tools necessary to the application of Natural Language Processing (NLP) and semantic search techniques. These tools were cobbled together exclusively from open source products freely available to everyone. They are delivered as services provided by artificial agents, because though I am passionate about NLP, my background is in Multi-Agent Systems (MAS).

docto.io

Applying NLP techniques to typeset documents (PDFs, DOCs, etc.) first requires extracting plain text. This is also true of images (GIFs, JPEGS, etc.), which require additional treatment via Optical Code Recognition (OCR). docto.io performs these extractions alongside general purpose file conversions. It even does this for images of paper documents stored in PDFs (for example).

By itself, docto.io is nothing special in terms of research and development. It is, however, a vital tool for those conducting research into NLP and semantic search. The best part is that it has no proprietary components. Everything of which it is comprised is entirely open source, including its OCR capabilities.

whatidid.info

whatidid.info is a domain I maintain to showcase product prototypes. At the time of writing, the software deployed there allows you to search disparate documents indexed from a variety of sources: i.e., typeset, web, and images of text. By itself it is powerful tool and a potentially viable cloud-based commercial product. Like docto.io, its constituent components are exclusively open source. Currently, it allows you to search a handful of documents obtained from the Alberta Hansard Office and a couple of business cards photographed and processed through my phone. Administrative access allows you to add any document you want from any source. whatidid.info is dynamic and changing, but the functionality described will be operational for the next couple of months so people can try it out.

The software behind whatidid.info is driven by an artificial agent. It is one of two agents in a two-member MAS. That is, all documents submitted by whatidid.info administrators are first processed by the agent behind docto.io. There is no limit to the number of agents that could be included in such an MAS. The technique behind this is documented in my masters thesis and has huge ramifications for administrative and bureaucratic roles, many of which could be fully automated.

The software hosted at whatidid.info is also significant because it provides the framework through which I would have implemented the new semantic search technique I had planned for the time remaining in my AITF funding period. Provided adequate business interest, I may still do this. One viable possibility is enabling people to search engineering log files in the oil and gas industry.

Gratitude

First, I would like to thank my boss, Marilynn Kalman of HireGround Software Solutions. It was a big mistake to terminate me while on parental leave. Not just because of the legal consequences, but because in all sincerity I wanted to make your company a success. I hope you do succeed and leave a lasting legacy that will provide for your staff for years to come. I thank you for this amazing, life-changing opportunity, which, if nothing else, showed me I can develop my own products. I don’t know what the future will bring, but it is my hope that I will never have to be an employee again (unless they pay me three times what you did).

Second, I thank the good folk over at Alberta Innovates Technology Futures. You made this all possible and I hope you agree that my time as an r&D Associate was a roaring success. (Why the wonky capitalization?) I have created tools that may benefit others given the same opportunity. I am slowly venturing into Calgary’s high-tech start-up community in search of contacts, talent, and mentors. I suspect I will cross paths with the agents of your organization and hope to engage you in the future, in whatever capacity.

Sincerely,

Daniel Bidulock, M.Div., M.Sc.

Yesterday I made some major improvements to the gebo-react-hai. This is a grunt-init template, which produces a human-agent interface for the gebo. gebo is evolving into a multi-agent system, and is an extension of my masters research. Currently, it primarily functions as a web application server. gebo-react-hai enables the human agent to send messages to the gebo agent.

Having built a similar HAI template in Angular, I thought I’d take a similar design approach. I like Angular and React, but they are different tools and so need to be wielded in different ways.

Yesterday I was approaching the gebo-react-hai in the way I had become accustomed working with Angular. In retrospect, I may have been doing Angular wrong anyway, so it was high-time I rethought my approach to React.

Here’s where this post gets boring. I gained no new philosophical insight into my craft, but instead confronted a nuts-and-bolts problem related to deployment. The gebo-react-hai template produces a project, which can then be developed, tested, and built with the help of grunt. When building, you can get grunt to do lots of handy things like concatenate and uglify your code. It was in the concatenation where everything got hairy.

The problem is this: HAI functionality is split between two pages (public and private, basically). After you authenticate and switch over to the members-only interface, you’re loading a whole new web page. Each page has the same basic dependencies, but each has their own custom React components. The purpose of concatenating, minifying, and all the rest is to produce a simple script, e.g.: my-project.min.js. If each page has its own unique dependencies, then there have to be two different deployment-ready files in production. And what if a HAI has even more pages?

Some people may roll like this, but it’s not my style.

Why do all that fancy grunt processing if you’re just going to have to rig a whole bunch of different minified files for each page that makes up your web application?

Why indeed? I was contemplating this very question driving my daughter to kindergarten when I realized that I should pay more attention to the hokey tech buzzwords that people are always dreaming up… single page app! Like duh. By putting all functionality on a single page, I don’t have to spend more time mastering grunt. Deployment is simplified and my software is easier to use for the developer. Again, like duh.

Now it’s time to see how the Bootstrap navbar and nav-tabs get along…