Service Provider and its underlying Mapping Concept

In short, the Service Provider is the microservice of the MTP platform which is responsible for executing functions using external APIs. Its responsibility is cleary decoupled from the Master Service which orchestrates the execution of a user routine (docu see Master Service Documentation). In the context of the overall microservice architecture of our prototype, the Service Provider is accessed solely via the Master Service, mainly to execute functions, which may have some input and may return some result. Besides retrieving all existing functions and APIs, the Service Provider allows to create, update or delete functions during runtime.

Since the Service Provider is so to say the implementation of our overall mapping concept for dynamic adding (i.e. during runtime) of new functions and external APIs to execute those functions. Therefore, in order to be able to understand the architecture of the Service Provider it is indispensable to understand the underlying Mapping Concept, i.e. the way how functions and APIs are represented, first. Thus, this documentation at first explains in detail how the underlying Mapping Concept works and afterwards focusses on the technical implementation of this concept by the Service Provider.

The structure of this documentation is as follows:

Underlying Mapping Concept

The motivation for the following concept stems from one of the core goals of this project, namely to enable the “execution of service compositions independent of the available API-endpoints”. Whereas the flexible composition of services is the responsibility of the Master Service, the latter part, i.e. the execution of individual services, which are represented by functions in terms of the Mapping Concept, is the responsibility of the Service Provider. Thereby, on the one hand, we wanted to be able to add new functions which then can be used as part of a routine. On the other hand, we wanted to be able to add new external providers, which are represented by APIs in terms of the Mapping Concept, which offer the functionality to execute the function by accessing certain endpoints of a external API. Both should be possible dynamically during runtime.

In order to understand the concept we developed to achieved these goals, it is easiest to take a look at the two data models of a function and an API one by one.

Data model - Functions

Attribute Type Description
user id Some id representing the user this function belongs to.
category String Area this function fits into. Has only a descriptive meaning and is used to structure the available functions by categories in the UI.
function_name String Internal name of the function. It is displayed in the developer view of the UI and must be unique under the functions of a given user.
function_label String Label for this function which is displayed in the UI when searching for available functions while creating or editing a routine.
result Object An object representing the result of this function if it has one. If a function has no result, this is respresented by an empty object. The attributes of a non empty result object are as follows:
Show/Hide structure
 
Attribute Type Description
name String Internal name of the result variable.
type String ("number", "text" or "boolean") The datatype of the result variable. Used for validation of the results from API calls.
label String Label for the result variable which is displayed in the UI when this function is to be used as part of a routine.
pattern String Optional. A valid Regular Expression for the Python library re (docu see here). Also used for validation of the results from API calls if present.
help_text String Optional. Detailed explanation of the result of this function.
fields List List of the fields (or parameters) that may or have to be passed to this function when it should be executed. If a function has no fields, this should be an empty list. The attributes of each field in the list are as follows:
Show/Hide structure
 
Attribute Type Description
name String Internal name of this field. Used within API specifications.
type String ("number", "text" or "boolean") The datatype of this. Used for validation.
label String Label for this field which is displayed in the UI when this function is to be used as part of a routine.
required Boolean If set to True this field must be specified when this function is invoked, otherwise not.
help_text String Optional. Detailed explanation of this field.

Data model - APIs

Attribute Type Description
user id Some id representing the user this API belongs to.
function_name String Name of the function this API belongs to, i.e. the value of the field function_name of the corresponding function.
name String Internal name of the API. It is displayed in the developer view of the UI and must be unique under the APIs of a given user.
url String (must be valid URL) URL to which the request is made when this API is accessed. May contain placeholders (see below).
header Object Header that should be sent together with the request to the API. Must be a valid JSON Object (might be empty).
request_params_template Object Parameters that should be sent together with the request to the API. Must be a valid JSON Object (might be empty). May contain placeholders (see below).
request_body_template Object Body that should be sent together with the request to the API depending on request method. Must be a valid JSON Object (might be empty). May contain placeholders (see below).
response_result_path String The path from which the value of the functions's result variable must be extracted from the API's response JSON after a successful call. Should be an empty string in case no result should be extracted. May contain placeholders (see below). Must comply with the following syntax: <key1>.<key2>.<key3>[<index>].<key4>, where key3 holds a list. More formally, it must match the following Regular Expression: ((\[\d+\])+(\.\w+(\[\d+\])*)*)|(\w+(\[\d+\])*(\.\w+(\[\d+\])*)*).
request_method String ("GET", "POST", "PUT", "PATCH", "DELETE") Request method to be used when requesting the API.
priority Number (0,1,2,3) The priority of this API which will be considered in the selection process when a function in invoked
  • "0" corresponds to "low"
  • "1" corresponds to "medium"
  • "2" corresponds to "high"
  • "3" corresponds to "preferred"
In case there are several APIs that can be applied to execute a given function, APIs with higher priority are chosen first. Only if none of the APIs with high priority is applicable resp. available, APIs with a lower priority class will be chosen. Priority "3" can be assigned only to one API for a given function and if there is any API at all there must be one preferred API. This preferred API determines which fields of the function are defined as required fields, namely, those fields that are required to apply this API.
enabled Boolean If set to True this API will be considered in the selection process, otherwise not.
placeholders List List of placeholders used in prior attributes that need to be replaced by "real" values before the API is accessed. If no placeholders were used, this is respresented by an empty list. The attributes of each placeholder object are as follows:
Show/Hide structure
 
Attribute Type Description
id number Id of this placeholder. Used to match it with it's usages in the aforementioned attributes. A placeholder is inserted into a template (JSON file or String) via §<id>§. At exactly this position the value of the placeholder is put when the template is filled (see later).
value Object Object which specifies how the placeholder shall be evaluated, i.e. how its value must be derived by potentialy using (some of) the fields of the invoked function specified by the user. There are two types of placeholder values: Those which correspond directly to one of the specified fields and those which are the result of another function that needs to be invoked first. In the latter case the specified fields may be used as parameters of this function.
The structure of the value object in either cases is shown in <a href=#appendix-1---exemplary-fixtures-of-function-and-api-specifications>Appendix 1</a> (For case 1 see step 2) and for case 2 see step 4)).
In both cases a boolean attribute apply_function specifies in which case we are. This information could also be derived from the structure of the object, but this attribute simplifies processing and validation. In the first case, i.e. if we do not apply a function, the name of the function field (attribute name) whose user specified value should be used as value of this placeholder is given in an attribute called field. In the latter case, besides the apply_function attribute, there are two more attributes: One, named function_name holding the internal name of the function to be invoked for evaluation. The second, named function_fields holding a list of objects specifying which fields are passed to the function for invocation together with their values. In case, a value is sourrounded by "§" signs, this means that this value should be the value of the corresponding field of the originally invoked function.
replace_as_string Boolean If set to True the value of the placeholder will always be converted to a string before it is inserted at the specified location.

The flow for the execution of a function with a given API can be roughly described as follows:

  1. Check, if all fields required for applying this API were specified.
  2. If so, evaluate all placeholders and store their values.
  3. Replace all placeholders in the corresponding attributes (request_params_template etc.) by their evaluated values.
  4. Make the actual API call using the specified URL, request method etc.
  5. If the call was successful and there is a result path specified, try to extract the result value from the response JSON according to specified path.

Technical Implementation

As already said, the Service Provider is a straightforward implementation of the Mapping Concept discussed in the prior section. Consequently the core tasks of the Service Provider are, one the hand, to manage a database of functions respectively API specifications and, on the other hand, to execute functions when requested. This is reflected in the three main endpoints the Service Provider provides:

Thus, at first the functionality of each endpoint is explained. After that, the flow of function execution including possible errors is depicted.

For the sake of completeness, it should be mentioned that the Service Provider has a fourth endpoint (/services/built_in_fixtures/) where a list of built-in function and API specifications can be fetched via a GET request containing a valid activation code as request parameter (/services/built_in_fixtures/?activation_code=<valid_code>). This specifications may be used to load some functions and APIs into a new user’s account, either for testing purposes or as an initial bundle of features.

Endpoints

Manage Functions Endpoint

This endpoint offers the possibility to create new functions, get a list of all existing functions as well as retrieve, update or delete an existing functions. In order to prevent as many errors as possible at this early stage all requests are validated exhaustively (see file function_serializer.py) before they are executed.

Manage APIs Endpoint

This endpoint offers the possibility to create new APIs, get a list of all existing APIs as well as retrieve, update or delete an existing API. In order to prevent as many errors as possible at this early stage all requests are validated exhaustively (see file api_serializer.py) before they are executed.

Invoke Function Endpoint

This endpoint offers the possibility to invoke a certain function by making a appropriate POST request.

If the request is valid (i.e. the function exists, all mandatory fields are specified, all specified fields have the correct type, etc.) the Service Provider tries to execute the function by using associated APIs. This process is discussed in the next section.

Flow of Function execution

When executing a function, at first, all available APIs for the given function are selected and ordered according to the selection logic described above (see description of attribute priority). Then a method called invoke_api (see file api_utils.py and Appendix 2) is executed one after the other for all APIs until an execution is successful. As parameters, the method is basically told which API is to be used and which function fields are given including the specified values: 1) Check if the given API is applicable, i.e. if all fields mandatory to use this API, i.e. to evaluate its placeholders, are specified 2) Evaluate all placeholders and store their values 3) Fill all templates belonging to attributes which required for making the request and the result extraction (url, header, request_params_template, request_body_template, response_result_path) by replacing all placeholders (§{placeholderId}§) with the values that were computed in the prior step. 4) Perform the actual API call 5) Extract the result value from the response JSON returned by the API according to the response_result_path, if not empty, and validate the extracted result against the function specification (i.e. check type and pattern, if one specifed) before it is returned.

Since the service provider is very generic and in principle allows the addition of any functions and APIs, erros can still occur during the execution of the method despite all previous validation. The possible exceptions (see file exceptions.py), all of which inheriting from an exception called ApiInvocationError, that might be thrown by the invoke_api method are explained below:

Error Description
ApiNotApplicableError Raised when the specified inputs are not sufficient to use this API.
ApiRequestError Raised when the request to the external API itself lead to an exception.
ApiCallNotSuccessfulError Raised when the status code of the response object of the external API call is not 2xx.
InvalidResultPathError Raised when result couldn't be extracted from the API response according to the specified result path.
InvalidResponseBodyTypeError Raised when the response body does not contain valid JSON.
PlaceholderEvaluationError Raised when the evaluation of a placeholder failed.
ResultValidationError Raised when the validation of the extracted result fails, e.g. because it has the wrong type and cannot be converted into the desired type.

Appendix 1 - Exemplary Fixtures of Function and API Specifications

Initial Remark: Many more fixtures for POST requests of Functions and associated APIs along with exemplary function invocations can be found here. There you can find the fixtures of several exemplary functions resp. APIs:

All of the following representations refer to the scenario that we want to add a function named “stock_price”, which should allow to query the stock price of a company which is passed as a parameter.

1) At first we add the stock_price function via a POST request to the manage_functions endpoint:

{
	"category":"Finance",
	"function_name":"stock_price",
	"function_label":"Retrieve Stock Price of a Company",
	"result":{
		"name":"stock_price",
		"type":"text",
		"label":"Stock Price",
		"pattern":"\\d+([.]?\\d+)?"
	},
	"fields":[
		{
			"name":"company_symbol",
			"type":"text",
			"label":"Company Symbol",
			"required":false,
			"help_text":"Symbol of the company"
		}
	]
}

The function has one parameter which allows to specify the company of interest a via its stock symbol.

2) Now we add an API that allows to execute this function via a POST request to the manage_apis endpoint:

{
	"function_name":"stock_price",
	"name":"Yahoo Finance (stock/get-detail by Symbol)",
	"url":"https://apidojo-yahoo-finance-v1.p.rapidapi.com/stock/get-detail",
	"header":{
		"x-rapidapi-key":"<secret_api_key>",
		"x-rapidapi-host":"apidojo-yahoo-finance-v1.p.rapidapi.com"
	},
	"request_params_template":{
		"lang":"en",
		"region":"US",
		"symbol":"§1§"
	},
	"request_body_template":{

	},
	"response_result_path":"price.regularMarketOpen.raw",
	"request_method":"GET",
	"priority":3,
	"enabled":true,
	"placeholders":[
		{
			"id":1,
			"value":{
				"field":"company_symbol",
				"apply_function":false
			},
			"replace_as_string":true
		}
	]
}

3) In order to use this function more conveniently without looking up the stock symbol, we want to have the possibility to specify the company of interest also via its name. Therefore, we add another field to the function via a PUT request to the manage_functions endpoint:

{
	"category":"Finance",
	"function_label":"Retrieve Stock Price of a Company",
	"additional_fields":[
		{
			"name":"company_name",
			"type":"text",
			"label":"Company Name",
			"required":false,
			"help_text":"Name of the company"
		}
	]
}

4) Now we can use the same external API as before by transforming the company_name value to the stock symbol of the corresponding company by using another function called retrieve_company_symbol. The specification of this function is not depicted here but can be found here. Therefore, we add another API (which we set as the new preferred one) which allows to execute this function if the company name is given via a POST request to the manage_apis endpoint:

{
	"function_name":"stock_price",
	"name":"Yahoo Finance (stock/get-detail by Name)",
	"url":"https://apidojo-yahoo-finance-v1.p.rapidapi.com/stock/get-detail",
	"header":{
		"x-rapidapi-key":"<secret_api_key>",
		"x-rapidapi-host":"apidojo-yahoo-finance-v1.p.rapidapi.com"
	},
	"request_params_template":{
		"lang":"en",
		"region":"US",
		"symbol":"§1§"
	},
	"request_body_template":{

	},
	"response_result_path":"price.regularMarketOpen.raw",
	"request_method":"GET",
	"priority":3,
	"enabled":true,
	"placeholders":[
		{
			"id":1,
			"value":{
				"function_name":"retrieve_company_symbol",
				"apply_function":true,
				"function_fields":[
					{
						"name":"search_name",
						"value":"§company_name§"
					}
				]
			},
			"replace_as_string":true
		}
	]
}

As alredy mentioned this API uses a placeholder which is evaluated by executing another function.

5) Finally, we want to invoke our function which can be done via a POST request to the invoke_function endpoint:

{
	"function_name":"stock_price",
	"specified_fields":[
		{
			"name":"company_name",
			"value":"Apple"
		},
		{
			"name":"company_symbol",
			"value":"AAPL"
		}
	]
}

It is mentionable, that the specification of the company symbol is optional since our preferred API only requires the company name only the company name is mandatory. However, by specifying optional parameters, we increase the chance of successful function execution since we could also apply the API specified in step 2) if the preferred API is not available.

Appendix 2 - Source Code for main part of Function Execution

Source code of invoke_api method:

def invoke_api(api: Api, specified_fields, invocation_stack):
    """
    Tries to invoke the passed API with the specified fields.
    :param Api api: the services to be invoked
    :param list specified_fields: specified fields for function execution
    :param list invocation_stack: list of functions waiting for the result of this function invocation
                                  (maintained to detect cycles due to placeholder evaluations)
    :return result of the underlying function if the API call was successful
    :exception raises exception if the API can't be accessed (e.g. specified_fields not sufficient, API not available etc.)
    """
    # Append the current function to the invocation stack
    invocation_stack.append(api.function_name)

    # 1) Check if this API is applicable, i.e. if all fields required to use this API are specified
    specified_fields_names = []
    for field in specified_fields:
        specified_fields_names.append(field["name"])

    required_fields_names = api.required_fields_names()
    if not all(field_name in specified_fields_names for field_name in required_fields_names):
        missing_fields = list(set(required_fields_names) - set(specified_fields_names))
        raise ApiNotApplicableError(api_name=api.name, missing_fields=missing_fields)

    # 2) Evaluate all placeholders and store values in a dict composed of id-value pairs
    placeholder_values = {}
    for placeholder in api.placeholders:
        value = _evaluate_placeholder(invocation_stack, placeholder, specified_fields, api.user)
        if value is None:
            raise PlaceholderEvaluationError(api_name=api.name, placeholder_id=placeholder["id"])
        else:
            placeholder_values[placeholder["id"]] = value

    # 3) Fill all templates required for the request, i.e. replace all placeholders by the computed values
    request_url = _fill_template(api.url, placeholder_values)
    request_params = {}
    if api.request_params_template:
        request_params = _fill_template(api.request_params_template, placeholder_values)

    request_body = {}
    if api.request_body_template:
        request_body = _fill_template(api.request_body_template, placeholder_values)

    # 4) Make API Call
    try:
        if api.request_method == "GET":
            response = requests.get(url=request_url, headers=api.header, params=request_params)
        else:
            response = requests.request(api.request_method, url=request_url, headers=api.header, params=request_params,
                                        json=request_body)
    except Exception as ex:
        raise ApiRequestError(api_name=api.name, exception_string=str(ex))

    if not status.is_success(response.status_code):
        raise ApiCallNotSuccessfulError(api_name=api.name, response_string=str(response), response_text=response.text)

    # 5) Extract result value from response JSON according to the response_result_path if not empty
    if api.response_result_path != "":
        try:
            response_data = response.json()
        except ValueError:
            raise InvalidResponseBodyTypeError(api.name)
        else:
            try:
                response_result_path = _fill_template(api.response_result_path, placeholder_values)
                result = _extract_response(response_result_path, response_data)
            except ValueError:
                raise InvalidResultPathError(api_name=api.name, result_path=api.response_result_path)
            else:
                #  validate result using the information given in the function specification
                result = _validate_result(result, api.function.result.get("type"),
                                          api.function.result.get("pattern"))
                if result is None:
                    raise ResultValidationError(api_name=api.name)
    else:
        result = None

    return result