Hello World with HTTPs→JSON→SQL

Though fizz-buzz-like exercises are necessary at some early development stages, now it's time to try something more practical. Today we'll make some simple program that:

• talks to HTTPS server endpoint
• downloads and parses JSON
• and saves results into an SQL database

1. Create the app

Launch VSCode, File->OpenFolder and chose the Argentum directory:

• It's either a downloaded windows-demo directory
• Or output directory, if you built Argentum from sources.

Create a httpJsonDbDemo.ag file in the ag subdirectory.

2. Declare dependencies and import everything needed:

using sys { log, setMainObject }
using string;
using httpClient { get, Response }
using json { Parser }
using sqliteFfi { Sqlite }

3. Boilerplate

Our application performs asynchronous network requests and as such needs a global state to be alive for the whole application lifetime. In argentum this global state must be incapsulated in an object:

class App {          // Our App class
    db = Sqlite;     // it will hold a DB connection, 
}                    // we could add other global items here later
app = App;           // Let's create the app instance
setMainObject(app);  // and register it inside the Argentum runtime

4. Connect to the database

The Sqlite class has the open method, that takes a path to a DB file and a set of flags. In the future there'll be added some useful named constants, but for now 2 is read-write access. This open returns a bool indicating if its database opens successfully.

app.db.open("mydb.sqlite", 2) ?
   log("Connected to the DB!")

5. Make an HTTPS query

Let's replace the log in line 12 with HTTPS query, since it should be performed only if db opens successfully.

The httpClient has a get function, that performs GET requests (It also exports a Request class that allows to performs arbitrary requests with any headers and verbs, but all we need now is a simple get).

This get function requires a URL and a delegate - an Argentum callable value type that combines:

• a function entry point
• and a weak pointer to an object that plays the role of a function context

Delegate functions can access their context objects using this keyword (or directly access fields and methods)

Since delegates store weak pointers to context, when they are called, Argentum locks and checks this context pointer, and skips the call if its context object is dead. This makes delegates perfect fit of pub/sub patterns. BTW delegate can be asynchronously called across threads, this automatically launches function on the thread of its context object. In fact our delegate will be called from the http transport thread.

For debug/trace purpose and for future serialization, all delegate has to have names unique in their modules.

In our example we create a delegate attached to our app object. We name this delegate onFetched. Http client calls this delegate, with Response parameter. For now we take a byte array from the response body, create a text string out of it and print it to the console.

Since this http call concludes our application activity, we gracefully terminate our application by setting a optional-empty value as the main runtime object. This assignment destroys the previous application state and exits from the application.

For this tutorial purpose we'll make a call to a mock endpoint generously provided by beeceptor.com

app.db.open("mydb.sqlite", 2) ?
get("https://fake-json-api.mock.beeceptor.com/users", app.&onFetched(resp Response){
    log("Data fetched {resp.body.mkStr(0, resp.body.capacity())}");
    setMainObject(?App);
});

At this point we can compile and run our application

• ctrl+shift+B in VSCode
• or bin\run-release httpJsonDbDemo in console

6. Parse JSON

The resulting JSON contains an array of objects having id, name and email fields. Let's extract them.

• replace the log call with a Parser initialization
• call the getArr method on our parser and provide a lambda that will be called once per each array item.

This lambda should handle the array item object, extract its fields and store it somehow. Let for simplicity make it other way:

• prepare three local variables for id, name and email,
• parse JSON object with fetching its attributes to these local variables
• and print these variables to log.

This eliminates the necessity of having an intermediate object.

    json = Parser.init(resp.body.mkStr(0, resp.body.capacity()));
    json.getArr {
        id = 0;
        name = "";
        email = "";
        json.getObj {
            _=="id" ? id := int(json.getNum(0.0)) :
            _=="name" ? name := json.getStr("") :
            _=="email" ? email := json.getStr("");
        };
        log("{id} {name} {email} ")
    };    
    setMainObject(?App);

Launch app again and see the output:

1 Kirk Bernier Pablo16@hotmail.com
2 Joshua Lynch Noel51@gmail.com
3 Terrill Howell Cicero.Watsica51@hotmail.com
4 Mozell Emard Harold15@gmail.com

7. Store our data in a DB

Let's replace the log call in line 23 with this code:

        db.query("
            INSERT INTO "table"("id", "name", "avatar") values(?,?,?)
        ", 0).setInt(1, id)
            .setString(2, name)
            .setString(3, email)
            .execute{_}

Here we:

• using db field of the app object (we are in a delegate that is connected to the main app object)
• create a DB Query object having three parameters
• fill the parameters of this query with our data
• execute this query, ignoring its result (btw the {_} construction is an empty lambda with one parameter)

That's it. Our application:

• opens a SqLite database
• makes an HTTPS call
• parses the result
• saves the extracted data in a database

Let's make one small change - move DB Query creation out of json.getArr. because query should be precompiled once and not compiled individually for every JSON array item.

Full example, one piece

The final total listing is here:

using sys { log, setMainObject }
using string;
using httpClient { get, Response }
using json { Parser }
using sqliteFfi { Sqlite }

class App {
    db = Sqlite;
}
app = App;
setMainObject(app);

app.db.open("mydb.sqlite", 2) ?
get("https://fake-json-api.mock.beeceptor.com/users", app.&onFetched(resp Response){
    query = db.query("
        INSERT INTO "table"("id", "name", "avatar") values(?,?,?)
    ", 0);
    json = Parser.init(resp.body.mkStr(0, resp.body.capacity()));
    json.getArr {
        id = 0;
        name = "";
        email = "";
        json.getObj {
            _=="id"    ? id := int(json.getNum(0.0)) :
            _=="name"  ? name := json.getStr("") :
            _=="email" ? email := json.getStr("");
        };
        id !=0 && name !="" && email != "" ? query
            .setInt(1, id)
            .setString(2, name)
            .setString(3, email)
            .execute{_}
    };
    setMainObject(?App)
});

Interesting features:

• This example is just 35 lines of code
• It compiles to 55Kb standalone self-contained windows executable (yes, it uses curl and sqLite DLLs but nothing beyond this)
• It doesn't allocate anything but a couple of service objects and a buffer for received HTTP data, in contrast JSON DOM approach would allocate hundreds of thousands of objects
• It performs all IO-bound operations in asynchronous manner on separate threads, completely offloading main thread
• It can be easily modified to perform multiple queries in parallel
• It shows good practices and safety measures:
  • it checks DB connection upfront
  • it caches DB connection
  • it precompiles queries
  • it uses parameterized SQLs to protect from SQL injections
  • it double-quotes SQL tables and field names as per SqLite standard
  • it checks for JSON validity and skips records with omitted data fields
  • it doesn't depend on field order in incoming JSONs
  • it employs curl-multi to reduce usage of sockets to just one
• Despite being multithreaded, it has no data races and deadlocks, and application code doesn't do anything to achieve this (and can't do anything to break these guarantees)
• And also it never crashes and never leaks memory
• And these all didn't cost anything in matters of readability and simplicity:
  • it has almost no type declarations
  • it has no lifetime, nullability or thread-safety annotations
  • no fighting with borrow checker
  • no delegate-unsubscribing to help GC fight memory leaks
  • no closeable/finalizable/droppable to free sockets and DB connections
  • no hussle and bussle, just pure code, directly doing business logic