In the first part, we've described how to scrape COVID-19 statistics and save extracted  data to JSON or  CSV files.  

We are going to create a simple COVID API service layer to grab live statistics from https://www.worldometers.info/coronavirus/#countries website periodically.

HTML Endpoints serve and share parsed information with third-party applications/services after that.

Endpoints.

Simple HTML server written in Go handles 2 endpoints for us:

GET /v1 - List all COVID-19 cases.
GET /v1/{cntr} - Get COVID-19 cases for a specified country.

Available {cntr} values: world, USA, spain, Slovakia and etc.

We run our open API at
https://covid-19.dataflowkit.com/v1
https://covid-19.dataflowkit.com/v1/world

You may use it absolutely for free.

How does it work?

Some parts of the code are apparent, so we omit it here. Please refer to our public repository to get full code.

//Start HTTP server to handle API endpoints. 
func Start(cfg Config) *HTMLServer {
        ...
        // Omitted for brevity
        ...
    
        //Get COVID-19 cases for specified country.
        router.HandleFunc("/v1/{cntr}", covidHandler)
        //List all COVID-19 cases
        router.HandleFunc("/v1", covidHandler)
        
        ...
        // Omitted for brevity
        ...

	return &htmlServer
}

// runScheduler launches updateCovidStat func to pull
// updated information periodically (every hour)
func runScheduler(done chan struct{}) {
	gocron.Every(1).Hour().From(gocron.NextTick()).Do(updateCovidStat)
	for {
		select {
		case <-gocron.Start():
		case <-done:
			fmt.Println("Scheduler: Stopped")
			return
		}
	}
}

Let's pass coronaPayload.json  to the parse endpoint of Dataflow Scraper Kit API.

{
   "name":"COVID-19",
   "request":{
      "url":"https://www.worldometers.info/coronavirus/",
      "actions":[

      ],
      "type":"chrome"
   },
   "commonParent":"tr[role]",
   "fields":[
      {
         "name":"Country",
         "selector":"#main_table_countries_today thead+ tbody td:nth-child(1)",
         "attrs":[
            "text"
         ],
         "type":1,
         "filters":[
            {
               "name":"trim"
            }
         ]
      },
      {
         "name":"Total Cases",
         "selector":"#main_table_countries_today .sorting_1",
         "attrs":[
            "text"
         ],
         "type":1,
         "filters":[
            {
               "name":"trim"
            }
         ]
      },
      {
         "name":"New Cases",
         "selector":"#main_table_countries_today .sorting_1+ td",
         "attrs":[
            "text"
         ],
         "type":1,
         "filters":[
            {
               "name":"trim"
            }
         ]
      },
      {
         "name":"Total Deaths",
         "selector":"#main_table_countries_today .sorting_1~ td:nth-child(4)",
         "attrs":[
            "text"
         ],
         "type":1,
         "filters":[
            {
               "name":"trim"
            }
         ]
      },
      {
         "name":"New Deaths",
         "selector":"#main_table_countries_today thead+ tbody td:nth-child(5)",
         "attrs":[
            "text"
         ],
         "type":1,
         "filters":[
            {
               "name":"trim"
            }
         ]
      },
      {
         "name":"Total Recovered",
         "selector":"#main_table_countries_today thead+ tbody td:nth-child(6)",
         "attrs":[
            "text"
         ],
         "type":1,
         "filters":[
            {
               "name":"trim"
            }
         ]
      },
      {
         "name":"Active Cases",
         "selector":"#main_table_countries_today thead+ tbody td:nth-child(7)",
         "attrs":[
            "text"
         ],
         "type":1,
         "filters":[
            {
               "name":"trim"
            }
         ]
      }
   ],
   "format":"json"
}
COVID-19 JSON Payload 

We schedule func updateCovidStat() to launch periodically (every hour) to keep this information up to date.  

var (
    payloadFile   = "coronaPayload.json"
    //covidStatistics map keeps data returned by func updateCovidStat() 
    covidStatistics   []map[string]string
)

// updateCovidStat - send requests to DFK API
// DFK API pulls an actual COVID-19 data to covidStatistics map    
func updateCovidStat() {
    //Load Payload to request live stats from worldometers.info
	payload, err := ioutil.ReadFile(payloadFile)
    if err != nil {
		fmt.Printf("Failed to read payload file: %s", err.Error())
		return
	}
    //Send POST request to Dataflowkit Scraping API.
	response, err := http.Post("https://api.dataflowkit.com/v1/parse?api_key="+*apiKey, "application/json", bytes.NewReader(payload))
	if err != nil {
		fmt.Printf("Failed to post request to DFK Scraper API: %s", err.Error())
		return
	}
	defer response.Body.Close()
	if response.StatusCode != 200 {
		body, err := ioutil.ReadAll(response.Body)
		if err != nil {
			fmt.Printf("Failed to read respose body: %s", err.Error())
			return
		}
		fmt.Printf("Failed to get COVID-19 statistics. Server returned: %s", string(body))
		return
	}
    //StatusOk
	body, err := ioutil.ReadAll(response.Body)
	if err != nil {
		fmt.Printf("Failed read response body: %s", err.Error())
		return
	}
	err = json.Unmarshal(body, &covidStatistics)
	if err != nil {
		fmt.Printf("Failed unmarshal response into map: %s", err.Error())
		return
	}
}

func covidHandler(...) handles API endpoints.
covidStatistics variable keeps actual COVID cases grouped by country.

  • If no {cntr} parameter specified, the full dataset returned.
  • Specify {cntr} parameter to extract results for this country only.
// covidHandler handles API /v1 and /v1/{cntr} endpoints 
// covidStatistics variable is parsed here according to 
// passed {cntr} parameter
func covidHandler(w http.ResponseWriter, r *http.Request) {
	...
        // Omitted for brevity
    ...

	w.Header().Set("Content-Type", "application/json")

	vars := mux.Vars(r)
	country, ok := vars["cntr"]
    //return results for all coutries
	if !ok {
		writeResponse(w, covidStatistics)
		return
	}
    
    //return results for specified country
	countryStatistic := map[string]string{}
	for _, countryStatistic = range covidStatistics {
		if strings.ToLower(countryStatistic["Country_text"]) == strings.ToLower(country) {
			writeResponse(w, countryStatistic)
			return
		}
	}
    
    //If specifid country not found return the very first result (world)  
	fmt.Println("Not Found")
	countryStatistic = covidStatistics[0]
	writeResponse(w, countryStatistic)
}

How to build COVID-19 API Server?

Clone public repository from GitHub.

git clone https://github.com/slotix/COVID-19.git

Run the following command to build Go binary

cd COVID-19/covid-19-service && go build

Get a Free API Key.

Dataflow Kit API Key is required to Start the server. You can obtain it from the user dashboard after free registration.  Once you sign-up, we grant you free 1000 credits.

Go to https://account.dataflowkit.com and either use Facebook/Google login or register with your email.

Click on the "Log in"  button to  register with your Facebook or Google account. Or press the "Sign Up" link to register with your email.

Dataflow Kit. Login or register here to get Free API Key.

Now go to dashboard Settings  and Copy your API Key to clipboard.

Start Service and send requests.

Now we can run the Service and try to send requests.

./covid-19-service -a DFK-API-KEY
COVID-19 Service : started : Host=0.0.0.0:8008
 

Open http://0.0.0.0:8008/v1 in your web browser.

As you can see, a extended JSON containing an actual COVID data returned.

JSON formatted COVID-19 data 

Type curl command in another terminal window to check out the Service:

curl 0.0.0.0:8008/v1/world
{"Active Cases_text":"1,311,078","Country_text":"World","New Cases_text":"+67,094","New Deaths_text":"+5,104","Total Cases_text":"1,846,837","Total Deaths_text":"113,883","Total Recovered_text":"421,876"}
 

You can pipe json_pp at the end, so you have a prettier JSON response:

curl 0.0.0.0:8008/v1/world | json_pp 
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   237  100   237    0     0   1338      0 --:--:-- --:--:-- --:--:--  1338
{
   "New Cases_text" : "+62,892",
   "Total Cases_text" : "1,915,149",
   "Total Deaths_text" : "118,984",
   "New Deaths_text" : "+4,790",
   "Active Cases_text" : "1,354,933",
   "Last Update" : "2020-04-13 20:11",
   "Total Recovered_text" : "441,232",
   "Country_text" : "World"
}

 

But we have to consider an even better way to display information.  

In the next part, I'm going to show how to bring actual JSON data coming from API endpoints to HTML Widget and share it with others, as shown below.

Resources

Dataflow Kit open COVID-19 Free API:

https://covid-19.dataflowkit.com/v1
https://covid-19.dataflowkit.com/v1/world

Free Coronavirus (COVID-19) Widgets:

https://covid-19.dataflowkit.com/


Public Github repository with COVID-19 open API sources.

slotix/COVID-19
Coronavirus (COVID-19) open API. Contribute to slotix/COVID-19 development by creating an account on GitHub.

Source of  COVID-19 actual data.

Coronavirus Update (Live): 1,775,210 Cases and 108,544 Deaths from COVID-19 Virus Pandemic - Worldometer
Live statistics and coronavirus news tracking the number of confirmed cases, recovered patients, tests, and death toll due to the COVID-19 coronavirus from Wuhan, China. Coronavirus counter with new cases, deaths, and number of tests per 1 Million population. Historical data and info. Daily charts, …