10,000 sentences: an Android language learning application

I'm not especially good at language learning, but I can speak 3.5 languages and I love learning them. When learning a language, there are many things to practice: grammar, pronunciation, listening comprehension, vocabulary, culture... And many applications try hard to make you learn all of them, but I think that's impossible. At the end of the day, you need a native speaker to practice. But there are some things where an application can help.

In the last year I thought a lot about language learning applications. In the process I tried to implement some ideas, and I'll present here my latest one. It is called 10,000 sentences and it is an Android application to help people learn new words in their target languages. The idea is based on an (unscientific) theory that you need 10,000 sentences to really learn a language. That may not be true, but if you really go through enough sentences -- it won't hurt and I think you will at least retain some new words and get an idea how sentences are formed. And that's what this application is about.

The application is just launched on Google Play: 10,000 sentences. It is open source (Apache 2.0), the source is available here. The sentences are exported (and prepared) from the Tatoeba languages dataset.

So... First of all, select the language you want to practice. There are 24 languages in both "directions" (so, for example there is "English for French speakers" and "French for English speakers):

Next, download the sentence for a language you learn. Sentences are ordered by complexity, the first sentences will be very short with simple words. For each sentence you will have a small quiz where you need to guess the words:

Note the translate widget in the upper right corner. You need to enable "Tap to translate" in Google Translate for this to work.

Also, the app will use text-to-speech if it's available for your language. Note that not all languages are available on all Android phones, but there are some TTS engines for various languages on Google play store.

If you fail in guessing the words -- the sentence will be shown again later. Otherwise, you can go to the next sentence.

If/when you encounter a new word, you can add it to your local dictionary (you can annotate it):

One annotation can have more than one words (for example the annotation "walk" will have the words "camminare", "camminavo", "camminero" in Italian). If you really succeed in going through all those 10,000 sentences at the end you will have your private dictionary with thousands of words. And if you use any of the memory cards app to practice (only) vocabulary you can export words from your dictionary to those apps.

PS. Of the 3.5 languages, Slovenian is the "0.5" one :)

Golang, appengine and unsafe imports

This post explains a common problem with Golang on appengine, but I still have no good fix for this problem. I'll leave it here just to explain it, in hope that sooner or later somebody will suggest a good solution in the comments section :)

On Google Cloud Platform (GCP) you have a few ways to use golang server-side. You can use appengine, the flexible environment (FE) or just plain VMs. Now, I think that their idea is to slowly move customers to the flexible environment, but I'm reluctant to do that for one reason -- deploying on FE is sloooow (for 5-10 minutes, comparing to 30s for appengine, including the start of the first instance).

In the long-term I definitely think I'll move to FE, because:

• The unsafe import problem (explained in this post)
• The current Appengine SDK works with go1.6, but future releases will (probably) lag behind new go releases by a few years. It was stuck with 1.4 for a long time.

But now I'm pretty happy with the current state of the Go support on Appengine. Except when one of my dependencies (or a dependency of a dependency) imports "unsafe". The unsafe package is not allowed on Appengine, and you get an error like this:

ERROR    2016-06-14 13:12:21,813 go_runtime.py:181] Failed to build Go application: (Executed command: /.../go_appengine/goroot/bin/go-app-builder -app_base /.../golang/src/... -arch 6 -dynamic -goroot /.../goroot -nobuild_files ^.*bytestream.go -unsafe -gopath /.../golang -print_extras_hash app.go)
2016/06/14 15:12:21 go-app-builder: Failed parsing input: parser: bad import "unsafe" in github.com/go-openapi/runtime/client_auth_info.go from GOPATH

Now, when this happens, first of all you will not get a correct filename where the import is. In my case it reported github.com/go-openapi/runtime/client_auth_info.go but the import was in github.com/go-openapi/runtime/text.go. So, you will get a random file in the same package when the import is.

What to do? Unfortunately, I have only two half-solutions. You can remove/change the dependency or you can try adding:

nobuild_files: ^github.com/go-openapi/runtime/text.go 

...in your app.yaml. And hope that the build will work without that file.

Any idea is welcome.

Simpler Golang reflection

It's not often that I need Golang reflection, but when I do... I always hated it. In Java you write something like obj.GetClass(), in Python it's obj.__class__ and you can get everything you need from there. And classes are basically descriptors for everything you can encounter in runtime (except primitives in Java).

In Golang, for every "thing", the medatata describing it is type. But there is also kind which sometimes you need to consult to get the real underlying type (if the kind is a pointer).

To get a list of fields, you need to check if it is a struct or a pointer to a struct. Then, when getting a list of fields, you must check if a field is anonymous or not. That's why you need a recursion to get the field list, but in that recursion some field names can happen twice (because they are declared in different anonymous structs).

Then to get/set a field by name, you need to check if the original is a struct or a pointer to a struct (again). If it is a struct you get the field value with ...FieldByName(fieldName) if it is a pointer it's ...Elem().FieldByName(fieldName),  but (only) for values you can use reflect.Indirect().  For every field you have two metadata types describing it: reflect.Value and reflect.StructField. In Java, when you have a reference to Field, you have everything you need to get/set/describe that field.

There is also a weirdness with nil values -- any nil value is not really nil in memory, but a nil with a type metadata. You can call (some) methods on nil values. So, in a way they behave like normal values, but you can't get/set their fields.

In short Java and Python reflection is much simpler than Golang reflection. At least, for me it is.

But, since sometimes I really had to do this reflection thing, I decided to simplify it as much as possible and created go-reflector. You basically give him any value, and it gives you a simple API to work with fields, methods and tags.

Example usage:

import "github.com/tkrajina/go-reflector/reflector"

p := Person{}

// Initializing:
obj := reflector.New(p)

// Fields:
obj.Field("Name").IsValid()
val, err := obj.Field("Name").Get()
err := obj.Field("Name").Set("Something")

// Tags:
jsonTag := obj.Field("Name").Tag("json")
jsonTag := obj.Field("Name").TagExpanded("json")
fieldTagsMap := obj.Field("Name").Tags()

// Listing fields:
fields := obj.FieldsAll()
fields := obj.FieldsFlattened()
fields := obj.Fields()
doubleDeclaredFields := obj.FindDoubleFields()
if len(doubleDeclaredFields) > 0 {
    fmt.Println("Detected multiple fields with same name:", doubleDeclaredFields)
}

// Calling methods:
resp, err := obj.Method("Hi").Call("John", "Smith")
if err != nil {
    fmt.Println("Cannot call method:", err.Error())
}
if resp.IsError() {
    fmt.Println("Method returned an error:", resp.Error.Error())
} else {
    fmt.Println("Method call response:", resp.Result)
}

// Get all methods:
for _, method := range obj.Methods() {
    fmt.Println("Method", method.Name(), "with input types", method.InTypes(), "and output types", method.OutTypes())
}

Read the README here: https://github.com/tkrajina/go-reflector.

ftmpl: a fast and typesafe golang (html) templating library

I always wondered why so many templating "languages" expect you learn a completely new language. For example, jstl, django templating, and golang templating. I know what a mess PHP can become if you mix too much PHP code, html and javascript code. But I, also, think that most developers today know that this is a no-no.

Another thing I dislike about templating libraries is that so many of them are type unsafe. Even when the original language is typesafe.

For example, in Golang:

type Inventory struct {
 Material string
 Count    uint
}
sweaters := Inventory{"wool", 17}
tmpl, err := template.New("test").Parse("{{.Count}} items are made of {{.Material}}")
if err != nil { panic(err) }
err = tmpl.Execute(os.Stdout, sweaters)
if err != nil { panic(err) }

But what if you call:

err = tmpl.Execute(os.Stdout, "jkljkl")

The template expects an argument that has Material and Count fields. But I executed it with a string! This will fail, but -- not in compile time! Keep in mind that in most cases templates are created in advance (not in runtime). This error should be caught in compile time, if you ask me. Especially if you (like me) moved to Golang from a scripting language -- because of its type safety.

Fast forward to my ftmpl... Ftmpl is a fast, typesafe templating for Golang I created for my last side project (more soon). Ftmpl generates Golang code from .tmpl files. Every template is "converted" to a Golang function, then you just call that function. If you declare your template with an argument Inventory then you can't call it with a string. The error will be caught on compile-time not in runtime!

See the README for more niceties (like extending templates, autoescaping, formatting output, ...).

My experience with golang

I rewrote a big part of Trackprofiler, my online GPS track editor. The web application is still mostly Python, the main difference is that all the GPS tracks manipulations (GPX files) are done in a microservice written in Golang. The main reason is that Python proved to be too slow and memory hungry for big GPX files.

I'll list a few things I learned in the process of rewriting it to Golang:

First, I want to emphasise that learning Golang has a lot less "WTF moments" than learning other languges.

My project has cca 9000 lines of code. I think that equivalent Python code is shorter, but if you write unit tests (you should), a lot of the Python ones will test things which in Golang aren't needed (because because they are caught by the compiler!).

The golang code tend to have more (but shorter) lines because in Golang you don't write things like:

something.DoSomething().DoSomethingElse().ThirdOperation()

Any of the .DoSomething(), DoSomethingElse(), ... calls can "throw" an exception. In many languages you just expect that somebody else will catch those exceptions. In Golang there is no "throw an exception". If there is an error you need to handle it immediately. The result is that you write every operation in a new line, and not cram them all in one.

But that's not necessarily bad, you have more lines, but they are all simple operations and easier to read and understand.

I thought that a big problem would be no DOM-based XML parser, but I love the builtin JSON and XML APIs!

Versioning dependencies sucks.

There are a lot of libraries^*, but most are not mature enough. This will probably change with time.

And, yeah, it has no generics.

^*If you are interested, here are my own open source libraries and utilities:

• gpxgo a GPX parser and manipulation library.
• go-elevations SRTM parser (gives you elevation for geo locations)
• ftmpl a fast, typesafe and compiled templating library
• Typescriptify golang structs convert golang structs into Golang classes

typescriptify golang structs

I'm a big fan of Go(lang).

I know, I know, Go don't have generics. I miss them, too. But on the other side -- it's simple, it's fast, vim go is great. Writing code is straightforward, it's readable. Golang is more like a better C (with a more consistent API and a garbage collection) with a feeling of a scripting language. It is not a "better Java".

Another language I (started to) like is Typescript. Not because I especially like the language or its syntax. It's because it is:

• typesafe,
• compiled to Javascript and the resulting code is similar (almost line-to-line) to the original Typescript code (making debugging very easy).

In my latest side project I decided to go the Go+Typescript way. There was only one thing missing. It is a web application with a lot of Javascript (i.e. Typescript code), but the data used in Typescript was not typesafe. On the server, the data is represented with Golang structs, but Typescript receives only their JSON representation. 

I solved the problem by writing a library which converts Golang structs to Typescript classes: github.com/tkrajina/typescriptify-golang-structs. 

It's simple, you just:

tscriptify -package=package/with/your/models -target=target_ts_file.ts Model1 Model2

....and then include target_ts_file.ts in your TypeScript scripts.

Make this step part of your bild process and... Now, if you change a field name in Golang -- the javascript/TypeScript compile will fail!

Golang GPX library

My golang port of gpxpy is finished, it's called (guess, guess...) gpxgo.

The first version of the port was done by Peter Vasil and I decided to continue from there. But my main aim was to implement a complete GPX 1.0 and GPX 1.1 specification. But, since his library was based on an older version of gpxpy, it was difficult to "reconcile" the different object models --so, I decided to change the library API. And this is how a new library is born.

Usage example:

import (
    ...
    "github.com/tkrajina/gpxgo/gpx"
    ...
)

gpxBytes := ...
gpxFile, err := gpx.ParseBytes(gpxBytes)
if err != nil {
    ...
}

// Analyize/manipulate your track data here...
for _, track := range gpxFile.Tracks {
    for _, segment := range track.Segments {
        for _, point := range segment.Points {
            fmt.Print(point)
        }
    }
}

// (Check the API for GPX manipulation and analyzing utility methods)

// When ready, you can write the resulting GPX file:
xmlBytes, err := gpxFile.ToXml(gpx.ToXmlParams{Version: "1.1", Indent: true})
...

Like gpxpy, gpxgo includes a command line utility to get some basic stats from .gpx files:

$ gpxinfo Mojstrovka.gpx
File: /Users/puzz/golang/src/github.com/tkrajina/gpxgo/test_files/Mojstrovka.gpx
GPX name:
GPX desctiption:
GPX version: 1.0
Author:
Email:


Global stats:
 Points: 184
 Length 2D: 2.6958067369682577
 Length 3D: 3.00439590990862
 Bounds: 46.430350, 46.435641, 13.738842, 13.748333
 Moving time: 0
 Stopped time: 0
 Max speed: 0.000000m/s = 0.000000km/h
 Total uphill: 446.4893280000001
 Total downhill: 417.65524800000026
 Started: 1901-12-13 20:45:52 +0000 UTC
 Ended: 1901-12-13 20:45:52 +0000 UTC


Track #1:
     Points: 184
     Length 2D: 2.6958067369682577
     Length 3D: 3.00439590990862
     Bounds: 46.430350, 46.435641, 13.738842, 13.748333
     Moving time: 0
     Stopped time: 0
     Max speed: 0.000000m/s = 0.000000km/h
     Total uphill: 446.4893280000001
...etc...