Making JSON more readable in the Terminal with Curl extension jq

jq banner

When developing apps and games you may need to read JSON files quickly in the Terminal, this can be a pain.

JSON is a right pain to read in the Mac terminal but Cloudant have some great tips for viewing JSON here.

First of all here is a random JSON file displayed in the mac terminal using curl.

Terminal Command:

1
curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5'

json default

What a mess.

Solution

Install the ./jq lightweight command line JSON processor from http://stedolan.github.io/jq/

Install (type this in the terminal and install)

1
2
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install jq

fyi: Homebrew is a packaged installer (read more http://brew.sh/) and check out http://braumeister.org/ to see what other packages you can install.

Conclusion

Now lets try the curl command again with ” ! jq .” added to the end of the curl request.

Terminal Command:

1
curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq .

color coded json in terminal

If you just want a web based tool try the Postman extension for Google Chrome.

Simple lua function to catch a nil when checking json values

Simple lua function to catch a nil when checking json values

function IsInString(theString, theText)
    print("IsInString (theString: '" .. theString .. "', theText: '" .. theText .. "').");
    if (tonumber(string.find(theString, theText)) == nil) then
        print(" String Not Found.");
        return false;
    else
        -- check the pos if the string to confirm (not nil is not enough)
        if (tonumber(string.find(theString, theText)) > 1) then
            print(" String Found.");
            return true;
        else
            print(" String Not Found.");
            return false;
        end
    end
end

I was doing this when processing json data but kept getting nil in the second elseif (nil was not being caught higher up.

if (event.response == nil) then
            --code
        elseif (event.response == "") then
            -- code
 
        elseif (tonumber(string.find(event.response, "ok")) > 1) then
            if (string.find(event.response, "true") > 1) then
                -- code
 
            end
            ...

Calculating the distance and bearing between two GPS points

I have been searching for a solution to calculate the distance and bearing of a target gps location for a while now. This page always popped up but my brain hurt every time I tried to convert this to lua.

I am happy to say I have cracked it and here is my code.

function Geo_Angle( lat1, lon1, lat2, lon2)
  local dLon = math.rad(lon2-lon1);
  local y = math.sin(dLon) * math.cos(math.rad(lat2));
  local x = math.cos(math.rad(lat1)) * math.sin(math.rad(lat2)) - math.sin(math.rad(lat1)) * math.cos(math.rad(lat2)) * math.cos(dLon);
  local brng = math.deg(math.atan2(y, x));
  return ((brng + 360) % 360);
end

The distance code was created by roaminggamer else on the Corona forums (can’t find their name).

 
function Geo_Distance(lat1, lon1, lat2, lon2)
  if lat1 == nil or lon1 == nil or lat2 == nil or lon2 == nil then
    return nil
  end
  local dlat = math.rad(lat2-lat1)
  local dlon = math.rad(lon2-lon1)
  local sin_dlat = math.sin(dlat/2)
  local sin_dlon = math.sin(dlon/2)
  local a = sin_dlat * sin_dlat + math.cos(math.rad(lat1)) * math.cos(math.rad(lat2)) * sin_dlon * sin_dlon
  local c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a))
  -- 6378 km is the earth's radius at the equator.
  -- 6357 km would be the radius at the poles (earth isn't a perfect circle).
  -- Thus, high latitude distances will be slightly overestimated
  -- To get miles, use 3963 as the constant (equator again)
  local d = 6378 * c
  return d
end

Here is how I call the code

print("Bearing between Tamworth and Armidale: " .. tonumber(GeoAngle(-31.1063157, 150.9307341,-30.5143425, 151.66696439999998)))
print("Distance between Tamworth and Armidale: " .. tonumber(GeoDistance(-31.1063157, 150.9307341,-30.5143425, 151.66696439999998)) .. "km")

Results:

Corona Simulator[4155:693467] Bearing between Tamworth and Armidale: 47.07677793374
Corona Simulator[4155:693467] Distance between Tamworth and Armidale: 96.419986257454km

When comparing my results with this site my bearing results are within 0.18677793374 degree and 0.078686257454 km for distance (good enough for my simple augmented reality app).

I tried to trick it by putting the source/dest either side of GMT and it did not measure the longest route around the globe.

Corona Simulator[4288:936166] Bearing between -179 lon and 179 lon: 270
Corona Simulator[4288:936166] Distance between -179 lon and 179 lon: 222.6341993844km

Also putting a source/dest near each pole calculated the approximate right distance between the coordinates.

Corona Simulator[4288:936166] Bearing between -89 lat and 89 lat: 180°
Corona Simulator[4288:936166] Distance between 89 lat and 89 lat: 19814.443745211km

Saving to Cloudant DBaaS from Corona SDK

Following on from my Cloudant DBaaS 101 post here is how you can save to a Cloudant DBaaS from Corona Labs SDK iOS/Android app.

Just raw code. Needs work but gives you a way to have a reusable function that also has a network listener to read the response from cloudant.

-- Require Objects
local socket = require("socket");
local http = require("socket.http");
local ltn12 = require("ltn12");
local json = require ("json");
local mime = require( "mime" );

-- The json data variable
jsondata = {}

-- Alert Box Handler
local function onGenericAlertBox( event )
if "clicked" == event.action then
local i = event.index
if 1 == i then
end
end
return true
end

-- The Network Save Listener
local function CloudantNetworkListener( event )
-- You function needs to be bigger than this but this gives you an idea how to process data back from Cloudant.
local retval
local errorCaptured = false
if ( event.isError ) then
if errorCaptured == false then
--ERROR: Error during request, code: -1003, details: A server with the specified hostname could not be found.
print ( "NetworkError:isError RESPONSE: " .. event.response )
retval = native.showAlert( "Error submitting your event", "This is embarrassing. There was an error saving your request. Here is the raw error: " .. event.response, { "OK" }, onGenericAlertBox)
errorCaptured = true
end

else
DoPrint("." .. event.response .. ".")
if (event.response == nil) then
-- show network down error

elseif (event.response == "") then
-- show network down error

elseif (string.find(event.response, "ok") > 1) then
if (string.find(event.response, "true") > 1) then
print ( "NetworkError: RESPONSE: " .. event.response )
retval = native.showAlert( "Saved", event.response , { "OK" }, onGenericAlertBox)
errorCaptured = true

end
elseif (string.find(event.response, "error") > 1) then
--{"error":"unauthorized","reason":"one of _writer, _creator is required for this request"}
if (string.find(event.response, "unauthorized") > 1) then
DoPrint ( "NetworkError: RESPONSE: " .. event.response )
retval = native.showAlert( "Error submitting your event", "This is embarrassing. I was unable to save your event due to invalid access." , { "OK" }, onGenericAlertBox)
errorCaptured = true
end
end

if errorCaptured == false then
print ( "NetworkError: RESPONSE: " .. event.response )
retval = native.showAlert( "Error submitting your event", "This is embarassing. There was an unknown error saving your request. Here is the raw error: " .. event.response, { "OK" }, onGenericAlertBox)
else
-- Save OK, Carry On
end
end
end

function SaveToCloudant(url, action, in_json, callback)

local json = require ( "json" )
local headers = {}

headers["Authorization"] = "Basic " .. mime.b64( "yourusername:yourpassword" )
headers["Content-Type"] = "application/json"

local params = {}
params.headers = headers
params.body = json.encode( in_json )

network.request ( url, action, callback, params )
end

-- The Data To Save
jsondata.nickname ="simon"
jsondata.sample_string = "yada"
jsondata.sample_int = 123
jsondata.datetime = tostring(os.time())

-- Save to Cloudant
SaveToCloudant("https://youraccount.cloudant.com/testtable", "POST", jsondata, CloudantNetworkListener)

Cloudant DBaaS 101

I have been playing with http://www.cloudant.com DBaaS and I am very impressed. If you want a NoSQL scalable database goto https://cloudant.com/sign-up/ and create free account. They have reasonable rates once you go beyond the free tier but you will never get a bill if you use less than $50 a month (approx 1,600,000 light reads or 300,000 heavy writes) or 50GB. Cloudant can handle billions of transactions a day, replicate data across data centres, secure/encrypt your data etc

The hardest part is understanding how to prepare HTTP requests to the server (and formatting/reading the JSON payloads) but that is way simple than dealing with the hell of SQL and HTTP forms, sessions and SQL injection stuff. When you signup choose your name (e.g “testuser”) this will become the URL to your database (“https://testuser.cloudant.com”). You can also choose where your database is located (US, London, Singapore etc) but don’t worry, this can be changed later. By default all traffic to/from cloudant is via https (at no cost) unless you opt out.

If you are unsure just goto https://cloudant.com/sign-up/ and wait 5 min and a popup live chat window will appear and ask away. You can also follow @cloudant on twitter, join the community on IRC or email support@cloudant.com.

My blog post is certainly not going to teach your the right terminology (for that go here https://cloudant.com/for-developers/ and https://docs.cloudant.com) but in a few minutes you can easily create a table, write new records, query and modify records in minutes.

How to create a table

Once you have created an account you can create tables (database) via the Cloudant url (https://{yourusername}.cloudant.com/dashboard.html) or remotely via a number of ways (e.g a Python Script). You do not need to use Python, you could use .NET, HTTP code from an Arduino or PHP code on a website. As long as you can send a http request with authorisation (login details) and attach/decode JSON you can talk to/from a Cloudant DBaaS.

import requests
import json
auth = ('yourusername', 'yourpassword')
post_url = "https://yourusername.cloudant.com/testtable".format(auth[0])
r = requests.put(post_url, auth=auth)
print json.dumps(r.json(), indent=1)

This will create table called “testtable”

If you do something wrong cloudant will let you know via the API and HTTP response code https://docs.cloudant.com/api/basics.html. Always check the response codes.

How to save a new record into a table

Obviously you need to decide on the table names and fields in your database table but here is a quick way to throw a record at the database above with python.

import requests
import json
doc = {'username':'kerrie','high_score':550, 'level':3}
auth = ('yourusername', 'yourpassword')
headers = {'Content-type': 'application/json'}
post_url = "https://yourusername.cloudant.com/testtable/kerrie".format(auth[0])
r = requests.put(post_url, auth=auth, headers=headers, data=json.dumps(doc))
print json.dumps(r.json(), indent=1)

You can see that we are creating a JSON array (containing some key pair values and passing that to a URL via HTTP packet data).

How to query a record

Simply query the URL (http://yourusername.cloudant.com/table/record

and process the JSON returned.

import requests
import json
auth = ('yourusername', 'yourpassword')
get_url = "https://yourusername.cloudant.com/testtable/kerrie".format(auth[0])
r = requests.get(get_url, auth=auth)
print json.dumps(r.json(), indent=1)

There is a whole topic on querying by ID’s and versioning that is not covered here.

How to query all record

You can also return all records in a table by querying the URL: https://yourusername.cloudant.com/testtable/_all_docs

Results would be returned something like this:

{"total_rows":4,"offset":0,"rows":[
{"id":"jane","key":"jane","value":{"rev":"5-t8ca55e2c81d38895ba8eff46a53c5db"}},
{"id":"kerrie","key":"kerrie","value":{"rev":"4-5e21e7e140af02afdc2f37b7538bcc34"}},
{"id":"marc","key":"marc","value":{"rev":"5-ha8a12fc9829baa09c5940edcr6fdf9f"}},
{"id":"simon","key":"simon","value":{"rev":"6-a9e10252931e4df63aa5ce055c56aac9"}}
]}

There is also syntax for maximum items returned and pagination that you can read up on.

Now that you understand the basics you can plan how you want to talk to your database, it could be via .NET, PHP, Perl, Corona SDK or via Python. Do check out the official create/read/update and delete examples and read the getting started guide.

In the next post I will add some CoronaSDK to Cloudant tips.

Edit 2/Jan/2015: Cloudant have increased the freebies 10 fold (I have updated the prices above).

Arduino GPS Code for GPS/GPRS/GSM Module V3.0 (SKU:TEL0051) loaded

Arduino GPS Code for GPS/GPRS/GSM Module V3.0 (SKU:TEL0051) loaded.

20140727-031035-11435603.jpg

http://www.dfrobot.com/wiki/index.php/GPS/GPRS/GSM_Module_V3.0_%28SKU:TEL0051%29#How_to_drive_the_GPS_Mode_via_USB_port

Quite a bit of research to get the boards code, configure the arduino IDE and latest GitHub code.

Now I have to go outside to get a GPS signal.

20140727-031152-11512745.jpg

Update: Was able to send a text message from the Arduino/gsm module Telstra pre paid sim via power pack alone. It turns out we need to redirect pins 3/4/5 to 11/12/13 🙂

20140727-162112-58872675.jpg

20140727-162734-59254900.jpg

Checking (and verifying) the internet on a raspberry pi with a bash script

This is how I check the internet with a bash script on a raspberry pi. It verifies the internet by downloading a file from a webserver (e.g http://www.your-domain/pi/test/simple_text_file.txt
(change this to a file of your choice)).

Script: CheckInternet.sh

    #!/bin/bash
    printf "BASH Version: $BASH_VERSION\n"

    printf "Testing Internet Connection (http://www.your-domain.com):\n\n"
    # ping -q -t 10 -w1 -c1 www.your-domain.com
    wget -q --tries=2 --timeout=20 http://www.your-domain.com
    if [[ $? -eq 0 ]]; then
    printf "Internet appears online.\n\n"
    TMP_FILE="./simple_text_file.txt"
    if [ -f $TMP_FILE ]
    then
    echo "Deleting old temp files.."
    sudo rm ./simple_text_file.txt
    fi
    printf "Downloading file from the internet.\n"
    sudo wget http://www.your-domain/pi/test/simple_text_file.txt
    if [ -f $TMP_FILE ]
    then
    printf "File downloaded ok.\n"
    printf "File Contents:\n"
    echo "----------------------------------------------------------"
    printf "\n"
    cat $TMP_FILE
    printf "\n\n"
    echo "----------------------------------------------------------"
    else
    printf "File failed to download.\n"
    fi
    else
    printf "Internet appears offline, unable to download files."
    fi

    printf "Tiding up..\n\n"
    sudo rm ./simple_text_file.txt

    printf "Goodbye\n\n"

To call this script from the command line do this.

    sudo bash CheckInternet.sh

Alternatively you can schedule emails by adding this to your “/etc/crontab” file


    # run every 15 minutes
    */15 * * * * root bash /usr/bin/simon/CheckInternet.sh

    # run every 5 minutes
    */5 * * * * root bash /usr/bin/simon/CheckInternet.sh

Or you can run this on pi start-up by adding the python call to your “/etc/rc.local” file.

    bash /usr/bin/simon/CheckInternet.sh

Make sure you make you file executable.

Send Email from Raspberry Pi

This is how I am sending emails (boot and 15 min runtime notifications) from my Pi

Python Script: /usr/bin/simon/SendEmail.py

You will need to type in your mail server Username/Password/IP/Port to get this to work.


    #!/usr/bin/python
    import smtplib

    EMAIL_USER = 'your-email@your-domain.com'
    EMAIL_PASS = 'your_email_password_goes_here'
    SMTP_SERVER = '192.0.0.1'
    SMTP_PORT = 587

    def send_email(recipient, subject, msg):
    smtpserver = smtplib.SMTP(SMTP_SERVER, SMTP_PORT)
    smtpserver.set_debuglevel(0)
    smtpserver.ehlo()
    smtpserver.starttls()
    smtpserver.login(EMAIL_USER, EMAIL_PASS)
    header = 'To:' + recipient + '\n' + 'From:' + EMAIL_USER + '\n' + 'Subject:' + subject + '\n'
    header = header + '\n' + msg + '\n\n'
    msg = header + '\n' + msg + '\n\n'
    smtpserver.sendmail(EMAIL_USER, recipient, msg)
    smtpserver.close()

    send_email('recipient-name@gmail.com','Subject Goes Here', 'Email contents goes here.')

To call this script from the command line do this.

    sudo python SendEmail.py

Alternatively you can schedule emails by adding this to your “/etc/crontab” file


    # run every 15 minutes
    */15 * * * * root python /usr/bin/simon/SendEmail.py

    # run every 5 minutes
    */5 * * * * root python /usr/bin/simon/SendEmail.py

Or you can run this on pi start-up by adding the python call to your “/etc/rc.local” file.

    python /usr/bin/simon/SendEmail.py