HIGH.FI - News aggregator API

This is a documentation for building web/mobile apps on top of HIGH.FI's news aggregator platform. HIGH.FI is a service that collects news headlines from thousands of news sites and organizes them in categories, gives each headline a score and provides that data in chronological order through an API.

Essentially, the service is similar to an RSS reader, but our staff has pre-selected the news sources, categorized them and the actual retrieval of the headlines (i.e. the heavy lifting) is done on HIGH.FI's servers. Thus, the API provides an end-point to get all that data from one single point.

Table of content

API version history

2018-09-18 Added shareWithTag field to news lists output. This suggests a hashtag that we recommend using alongside the headline and URI when sharing the news story via your service.

Applies to news lists
2018-03-16 Added paywall field to source listing, which indicates whether the news source is behind paywall or not.

Applies to source listing
2017-06-02 Added ampURL output variable for news lists output. This variable is available if the news URL has an Accelerated Mobile Page (AMP) version available. If so, the URL is a direct link to that AMP version of the news item. AMP pages should be used on environments where the rendering engine supports such pages and where speed of rendering is high priority in terms of usability of the service/app you are developing.

Variable is not available for items that don't have an AMP edition.

Applies to news lists
2017-04-10 Added availableUntil output variable for listCategories call. This variable tells, if the category is a custom one and regardless of your caching methods, should disappear completely after the set date/time. Used, for now, in ICYMI categories.

Applies to listCategories
2017-03-17 Made changes to the example workflow and mockup application flow parts to highlight the fact that server response for news lists already contains click-tracking URIs with the mandatory deviceID, so deviceID don't have to be added manually to the click-tracking URIs.
2017-03-03 Added a possibility to serve user-tailored categories via listCategories call. Added highlight return value to the listCategories call and added a note that the listCategories call shouldn't be cached for more than 30 minutes at a time.

Applies to listCategories
2017-03-03 From now on, we require passing of the appID, deviceID and APIKEY variables in the URLs when retrieving languages and lists too. This made has been change in order to be able to track better what categories people browse and when they were last online, in order to provide a summary of headlines they might have missed since last visit.

Applies to listing available languages, listing of news categories, listing available news sources and headline lists.

Made changes also to the example workflow to highlight the changes in requirements.
2016-05-25 Added a new field to news listing output, called highlightType that accompanies highlight field when set to "true", indicating the kind of highlighting the specific news headline requires.

Applies to news listing output.
2015-10-05 As per requested, added a mobile companion to shareURL output variable called mobileShareURL which will return the preferred URI for sharing purposes, if the page has a separate mobile version of it. If not, the variable will be empty and the app/service should use shareURL instead.

Applies to news listing output.
2015-10-02 Added new output variable to news listing, called clickTrackingLink, which is the URI that an app / service should call in the background after sending user to a specific news article via either originalURL or originalMobileURL. Replaces old link and mobileLink variables (that also do redirect after the click-tracking, which is pointless if the workflow is done as preferred).
Clarified the usage of link and mobileLink variables in news headline listing output. They are essentially heritage methods that will eventually be phased out / depreceated.
Changed the example mockup code to use the clickTrackingLink and originalURL correctly.
Added new variable to news listing called shareURL which is the recommended URI for sharing a specific article. It will allow us to track sharing of the articles that originate from HIGH.FI API.
Fixed a typo in example code flow that had two question marks (?) in the click tracking URI call (latter one should have been an ampersand (&) instead).
2015-06-18 By default, now all news lists omit paid-for / paywalled news sites. This includes sites that have limited number of freely readable articles per month. To show the sites that are blocked, add ?includePaid=true to your news list requests.
2015-04-04 Added new output variable to news lists, called originalPicture which links directly to the original picture found for the image. Note, the resolution, size nor the availability of that image cannot be guaranteed, as the URL points directly to the publisher's image server.
Changed news lists' picture output variable to be delivered from HIGH.FI's servers, and as a standardized 100x100px thumbnail rather than a full-blown hi-res image. If you need to use bigger sizes, use originalPicture variable instead.
2015-03-03 Added new method: listSources, which will provide full list of news sources available for each language.
Added sourceID output variable to news lists, which matches with the listSources news source list's ID values.
2014-11-21 Added Estonia as a new region.
2014-10-24 Added beta support for Italian news.
2014-09-26 Backend fix: Search results didn't include headline's mobile URL at all.
2014-08-12 Backend fix: In some cases, headline title could have contained non-standard (JSON) characters, such as linefeeds/carriage returns.
Backend fix: Removed carriage returns & linefeeds from description field and replaced with spaces.
Added a new output variable to news/headline lists called publishedDateJS which return JavaScript/toJSON -compliant date string. This also sets the previous publishedDate variable to depreciated state and we plan to end support for that variable by end of 2015 and hope that all apps will be converted to use the new variable instead.
2014-08-10 Fixed a backend bug, which caused news lists from 2nd page onwards to render invalid JSON (trailing comma after the last list item).
2014-08-06 Added two new return values to listLanguages: genericNewsURLPart and useToRetrieveLists.
Added pseudo-code/logical example, copyright info, etc to the API documentation.
2014-08-01 Initial public version opened.

Usage restrictions

As the API service is provided free-of-charge, we want to lay down some ground rules here:

Obtaining API key

To obtain an API key, contact us and let us know what kind of application you'd like to build and for what purpose. Remember to obtain a separate API key for each project.

Methods

The API itself is extremely simple, mostly read-only service that is used through plain HTTP service, returning UTF-8 JSON data. There are only handful of methods available and we're detailing them here.

listLanguages
listLanguages method is used without any parameters whatsoever. It simply results the list of currently supported by the server. You should cache this method's return value for min 24h, we don't add new languages that often :-)
Retrieval method GET
Required input variables API keyMandatory for all communication between the client and the server.
deviceIDoptional, however mandatory in order to allow personalization of headlines
appIDoptional, however extremely recommended in order to track application versions, etc
returns (JSON) responseData.supportedLanguages language Name of the language, in English.

example return value: Dutch
country Name of the country, in English. (i.e. the language and the country together build an unique identifier)

example return value: Germany
domainToUse To which domain the requests for that language should be pointed to. This is important, as the URL structure for lists is same for all languages, the domain to which the list retrievals are made to, identifies the language to the server. I.e. if this value is "xxx.yyy", lists for that language/country are retrieved from https://xxx.yyy/(the-api-url-part)

example return value: en.high.fi
languageCode Official language code/country combination, can be used to auto-detect the proper language according to user's preferences.

example return value: en-us
useToRetrieveLists Language/coutry combo in our own format, use this to retrieve lists to get the right lists.

example return value: english
mostPopularName Just for assistance, translated heading for "Most Popular" lists.

example return value: Suosituimmat
latestName Just like mostPopularName, this is the translated heading for generic "Latest news" lists.

example return value: Uusimmat uutiset
genericNewsURLPart This is a special "category" for listing all news. The value this field returns will be used to retrieve generic news lists (i.e. not from specific news category).

example return value: uutiset
listCategories
Returns full list of news categories available for the selected language. Note that the method doesn't return the always-present "Most popular" and "Latest news" lists, those are meant to be available always.

Note! You shouldn't cache this for more than 30 minutes at a time! This is due the fact that we can provide user-tailored "categories" in this listing that vary from time to time and those user-specific "categories" might or might not be available to the specific user at that time (such special cases can appear only when you use deviceID variable in your API requests).
Retrieval method GET
Required input variables usedLanguage Should be one of the useToRetrieveLists values returned by the listLanguages method. Should match the domain you're using (which is also specified by the listLanguages method)
API key Mandatory for all communication between the client and the server.
deviceID optional, however mandatory in order to allow personalization of headlines
appID optional, however recommended in order to track application versions, etc
Note that the domain you're using to call this method should match to the usedLanguage variable, i.e. if you're retrieving Finnish categories, your API call should go to high.fi domain and usedLanguage input variable should be set to finnish
returns (JSON) responseData.categories title Human readable name of the category.

example return value: Pohjois-Savo
highlight Boolean true|false.

If set to "true", indicates that this item should be highlighted somehow, ideally in similar way the always-existing "top" and "latest" categories are highlighted/shown. Typically highlighted categories are user-tailored categories that serve a special meaning for that particular user, only and might not be available to other users at all.
availableUntil JSON-compatible date/time, in UTC.

optional! - this variable is available only for selected categories and not even for those at all times.

Indicates for special categories when they will disappear / stop working. This is used for stuff like "In Case You Missed These" user-specific category. You know from this ´variable when to kill such category from listings when it appears.
sectionID Integer. Unique identifier for the category.
depth As the category tree is structured as an unlimited-depth tree, this determines at which depth this category is located at. Lowest possible depth is 1, in which case, it is a main category that is quite likely to contain sub-categories.
htmlFilename (Semi-)Unique identifier for the category, which has to be used in URLs when retrieving news listings for specific news categories. Note that this value is unique only to a specific language/country combination and must be accompanied with proper usage of domain when retrieving lists with this value.

example return value: lifestyle
parentSectionID If the category is at depth 1, i.e. is one of the main categories, this value will be set to zero (0). Otherwise, this value indicates a sectionID value, telling that the category is sub-category of parentSectionID.
listSources
Returns full list of news sources (sites) available for the selected language. Returned sourceIDs are also present in news lists, thus, allowing easy filtering and/or highlighting based on news source, if wanted to.
Retrieval method GET
Required input variables usedLanguage Should be one of the useToRetrieveLists values returned by the listLanguages method. Should match the domain you're using (which is also specified by the listLanguages method)
API key Mandatory for all communication between the client and the server.
deviceID optional, however mandatory in order to allow personalization of headlines
appID optional, however extremely recommended in order to track application versions, etc
Note that the domain you're using to call this method should match to the usedLanguage variable, i.e. if you're retrieving Swedish categories, your API call should go to sv.high.fi domain and usedLanguage input variable should be set to swedish
returns (JSON) responseData.newsSources sourceName Human readable name of the news source.

example return value: The New York Times
sourceID Integer. Unique identifier for the news source.
paywall ENUM. Returns one of the following options:
· free - totally free of paywall
· strict-paywall - all (or almost all) content is behind a paywall
· monthly-limit - there's a limit of free articles one can read on each month
· partial - some of the articles are behind paywall

Thus, if you want your project to show only totally free sources, use the free sites only.

Please note that you need to use includePaid option for news listing in order to retrieve any of the non-free news sources, they're all hidden from listings by default.

Retrieval of news lists

The actual retrieval of news lists is done through the site itself, not through API URL structure. Retrieval of specific category's news is done through HTTP request, formatted like this:

https://(listLanguages.domainToUse)/(listCategories.htmlFilename)/json-private?deviceID=(deviceID)&appID=(appID)&APIKEY=(APIKEY)

You can use optional input parameter when retrieving lists called jsonHideSections, which can be either empty or contain comma-separated list of sectionIDs that you want to be omitted from the list. Say, you wish to list all sports news, but omit ice-hockey (say, sectionID 123), you could call:

https://(listLanguages.domainToUse)/(listCategories.htmlFilename)/json-private?jsonHideSections=123&deviceID=(deviceID)&appID=(appID)&APIKEY=(APIKEY)

Pagination of news lists

For paging (retrieval of pages beyond the first one), append the page number before /json-private part, like this:

https://(listLanguages.domainToUse)/(listCategories.htmlFilename)/(page-number)/json-private?deviceID=(deviceID)&appID=(appID)&APIKEY=(APIKEY)

Note that each page contains max 70 headlines. If the previous page returned less than 70 headlines, the next page wont exist.

"Top news" & "All news" lists

There are two special cases available for retrieving news without specifying a specific category:

Return values for news lists

All data will be returned in JSON format. The variables returned are:

responseData feed title Used HIGH.FI locale's site title.
link Link to selected HIGH.FI locate's front page.
author Author behind the API service.
description Description for the returned JSON list.
type Returned format type. In this case, always json.
entries title The actual headline of the news story. always available
link This URL will redirect to your original URI and will track the click. Slower method to send the visitor to the news article and should be therefor avoided.

Note! Preferred method is to send the user to originalURL and do the click-tracking in the background, by doing a HTTP GET to clickTrackingLink. This variable will be eventually depreceated, but no decision on schedule has been made yet.

always available
clickTrackingLink URL your app needs to visit after a user clicks on the headline. This should be called in the background after the user has been sent to originalURL, to keep track of clicks. (mandatory method, we wont allow usage of the API without sending us the click-tracking data as the data will allow us to improve our selection)

Returns either 200 (tracking successful) or 404 (most likely non-existing article).

always available
author Site's name where the headline is from. If the site is located in sub-category, the subcategory name will be appended in parenthesis after the site's name. always available
publishedDate When the news article was published (actually, when the news article was added to HIGH.FI's database in most cases) always available

This variable has been depreciated since 2014-08-12 and will cease to exist by end of year 2015. Use publishedDateJS instead.
publishedDateJS When the news article was published (actually, when the news article was added to HIGH.FI's database in most cases) always available

example: 2012-04-23T18:25:43.511Z
picture If we have a 100x100 thumbnail version of the news picture available, its URL is located here. variable available only when picture is available
originalPicture If we have found an image for the news item - whether it is the favicon of the site or an actual picture related to the news item, the original URL of the image will be listed in this variable. Note tat these images can be slow and/or huge at times, so it is preferrable to use picture output variable instead. variable available only when picture is available
shortDescription Synopsis of the news story or abbreviation of it. variable available only if the site provides publicly visible abbreviation
originalURL URL of the story. Note! If you send your users to this URI, please do remember to do the background call to clickTrackingLink in order to allow us to keep track of API usage. always available
mobileLink Tracks the click and then redirects user to the mobile version of the article (if available, otherwise sends to the desktop/standard version).

Note! This method is will be depreceated eventually (no decision yet) and we prefer that you send the user directly to originalMobileURL (or in case that doesn't exist/is null, to originalURL instead) and do the click-tracking in the background by sending a single HTTP GET to clickTrackingLink.

variable always available, but if there's no separate mobile URL, this value will be empty
originalMobileURL If the site has separate mobile URL for the story, that mobile URL will be available in this variable.

variable always available, but if there's no separate mobile URL, this value will be empty
shareURL If your application / solution allows sharing the article to other users directly from your app/service, we recommend that you use this URI for sharing purposes. It might differ from the originalURL from time to time and might or might not include server-side tracking of shares.

variable always available
shareWithTag If your application / solution allows sharing the article to other users directly from your app/service, we recommend that you append this hashtag to the shared resource, say, after the headline text, but before the URI, in order to promote the ecosystem of ours.

variable always available
mobileShareURL If your application / solution allows sharing the article to other users directly from your app/service, we recommend that you use this URI for sharing purposes. If there's a separate mobile URI available for sharing, this variable will contain that – so, if you prefer to share mobile-friendly URIs, check this variable first and use it for sharing purposes.

Always available, but if the mobile URI is the same as "proper" URI, this will return an empty value and you should use shareURL instead.

It might differ from the originalMobileURL from time to time and might or might not include server-side tracking of shares.
ampURL If your application's rendering engine supports Accelerated Mobile Pages, you might want to use this URI, when available. Variable is available for headlines that have also an AMP version of the page available.

Only available when headline has an AMP version
articleID Unique identifier for the news article, integer. always available
sectionID Indicates to which category the news is associated to. Returns zero if the article is uncategorized. Corresponds to the sectionIDs listed by listCategories method. always available
sourceID Indicates the news source ID, corresponds to the sourceID returned by the listSources method.
highlight Boolean true|false. If set to "true", indicates that this item should be highlighted in the list somehow. Typically this being "true" indicates that the headline is currently in top70 most popular headlines chart. always available

See also highlightType to find out what the highlight flag can mean, in case you want to highlight different highlighted items differently, depending on their type.
highlighType If highlight is set to "true", this indicates why the specific line row should be highlighted. Current options are:
  • popular – indicates that the headline is currently one of the most popular ones. Should be highlighted in "generic" lists, like category listings, etc somehow.
  • advertisement – this is a paid-for advertisement. You can omit these items in your lists if you wish to. They provide revenue for maintaining HIGH.FI service, so we kindly ask you not to remove these, but we understand if you opt to do so.
  • collection – A bundle item that bundles several news stories under one link. These are highly customized links aimed for news junkies and are recommended to be highlighted, even promoted, in your app/service, as they typically provide a "big picture" to some specific news event, in one page.

Note! Process this field only if the highlight field is set to "true". This will return "none" for those rows that have the highlight set to "false".

Retrieval of news themselves / clicking on headline

Clicking on headline should follow this logic:

Typical workflow

To build a basic, multilingual news reader based on our API, the typical workflow of the app is somewhat simple. Here's our take on it:

  1. Application launches.
  2. Checks for (semi-)static/user-specific variables, whether they are already set.
  3. Retrieves list of available categories using the domain the user selected (in our case example, Dutch) by using URL https://nieuws2.nl/api/?act=listCategories&APIKEY=123&usedLanguage=dutch&deviceID=abcdefg12345&appID=(HIGH.FI for Tizen v0.1)
  4. Loads data to the default view, which in this app, is the generic latest news listing, using the URL https://nieuws2.nl/nieuws/json-private?APIKEY=123&deviceID=abcdefg12345&appID=(HIGH.FI for Tizen v0.1)
  5. User clicks a headline

Example mock-up app

Application launches

As the application launches, app looks if it already has a variable called apiKey available and whether it is empty or not. If it is empty, it retrieves a new key from developer's own servers, using a built-in URL:


Server returns API key


Now, the application stores this value to its permanent storage as apiKey.

Now the app looks if it has already generated a user-specific identifier. If not, it creates one and stores that identifier to userIdentifier variable. The data can be anything, as long as it fits into varchar(50) limits and is unique to each user and remains static, preferrably for entire lifetime of the application to that user. App now has stored the value to its internal variables:

Finally, app sets itself a version-specific identifier string to a specific variable. Say, appVersion, which fits into varchar(250). This would be now

Selecting the used language

App now checks if the user has selected her preferred news language by checking if userLanguage variable exists and is not empty. If not selected, app makes a HTTP call to retrieve list of supported languages from


The server then returns a list of supported languages in JSON data:


User then selects her preferred language, which in our case would be "English". App stores the following details into its variables:

Retrieve the list of available categories

As our example app has a list of available news categories shown on sidebar and the main list of news on other area, we will first pull the list of supported news categories to the app by making a HTTP call to


(note here the usage of useToRetrieveLists variable to fill the usedLanguage URL parameter and the usage of right domain, as per domainToUse variable, both stored in app's internal variables now)

This will return JSON:


This list of categories is then presented to the user. The order of the returned JSON is HIGH.FI's preferred order, i.e. not alphabetical one or ordered by the sectionID. You should also pay attention to the depth and (in case it being >1), the presence of parentSectionID variables, as those indicate that the item in the list is a sub-section of another section. The returned sectionIDs can be used later to omit categories, etc.

This list is now stored into local variable/scope by the app and presented as the list of categories. App caches this list for 30 minutes to avoid unnecessary API calls (but not for too long, in order to allow personalized categories to appear).

Retrieve the list of headlines

To show the actual news listing in the main panel, the app defaults to show the "all latest headlines" listing, a supercategory which wont appear in the previous category listing at all, but is always available and should always be available to the user. This listing is retrieved by making a HTTP call to the server:


This HTTP call will return a JSON:


App then presents this listing to the user in whatever fashion it prefers to do that. Of those headlines in the example, only the last one (the one from Gizmodo) appears to have the highlight set to true, meaning that the headline also appears in "top news" listing and should be somehow highlighted in other lists as "hot", if the app design is suited for such highlighting.

Also, only the news article from Fox has a news image associated to it, for other headlines, we don't know the news images' URL or there might not even be one for those articles at all.

As the listing is generic and none of the headlines appearing in this listing belong to this exact category (or in this case, a "supercategory") the category name is added to the name of the source in author variable.

User clicks a headline

Now, user decides to click the Fox headline. Our app will now launch an external browser window and send user to the originalURL value of the headline in previous JSON, to here:


In the background, the app also will make a silent HTTP GET request to the click tracking URL provided in the JSON's clickTrackingLink field. Now, the app adds the APIKEY and appID details to the URL before making the GET request. Thus, the HTTP GET will be send to URL:


Those two variables added to the URL were from app's internal variables, as set at the beginning of the application flow.

User removes/omits a specific news category

Now, our case example user doesn't care for tech news, at all. User wants to remove all news from Technology from her news lists. As we look at our previously retrieved list of categories, we can see that the Technology section's sectionID is 18. We will store this data to our internal variable scope:


a new category that needs to be hidden is added to app's internal variables

From now on, we use that variable in all of our headline list retrievals, whether those are "logical" or not (i.e. even when the app is retrieving headlines for "Sports" category, the list of omitted categories -- which now includes only "Technology" -- should be present in the URL).

An example retrieval of most popular headlines:


URL structure to retrieve most popular news from the server when one section has been omitted

Copyrights, limitation of liability, etc

HIGH.FI provides news headlines and their accompanying data (ingress, pictures, etc) from wide variety of news sources. None of the actual news material is provided and/or owned by us. In some legislations, collecting such data and presenting it in any kind of categorized way might (or might not) be against the local legislation. HIGH.FI is operated by AfterDawn Oy, a Finnish company and in our jurisdiction, providing a database of headlines, news abbreviation and thumbnails of news images falls under "citation rights" exemption in copyright legislation.

Usage of news pictures is tricky one, as many news outlets provide full-sized images alongside their newsfeeds -- and our servers simply hand out the URL pointing to those images, rather than caching of the images. In several legal cases around the world, the usage of "thumbnail-sized" versions of such images has been deemed to fall under "citation rights" exemption. Thus, it is advisable that you don't present the images in their full-size, even if available, but scale them down when presenting them in your application.

HIGH.FI simply provides the categorized data, but doesn't take any legal or moral responsibility of your app and/or app's actions nor legal status. If you feel you cannot use the API we provide because of these restrictions, then simply do not use it.

Public projects using HIGH.FI API

Here's a list of known and public projects using our API:

To-do

We have several things in our API to-do list, but here are the most urgent ones:

Contact us

We're eager to learn how you have used our API, so please, tell us! Also, if you have a great idea how you'd love to use the data, but we're missing some critical components from our API, contact us and tell us what you want. The API is a living organism and we'd love to improve it to suit for your needs.

Easiest way to contact us is to get in touch with us through our feedback form.