RVP is an application server to use to develop information systems.
-
Application server verifies the consistency and the integrity of the input attributes with the type system and the schema of the attributes.
-
Attributes in session database are used as identifying attributes in transactions and to implement roles and permissions.
-
Descriptive error messages help user to input the data in correct format with the needed dependencies.
The programming is reduced to implementing a type system of the attributes of the information system and to describing the dependencies of the attributes in the attribute configuration file. Adding transactions in URL:s is simple.
Introduction
Purpose
The application server is an automated application of database transactions. To develop applications in different purposes, the programming of new program code is not needed. The application server checks automatically the consistency and the integrity of the attributes of the input. Types can be configured with the regular expressions or by a reusable external module. These check the form of the attribute value. The model of the dependencies of the attributes is the relational model of the relational database management software. The model is configured by adding dependencies of the individual attributes. The model is used to check the consistency of the transaction attributes before the transaction. Configurable text strings are used to print messages in the output files to help the user to fill in the data in correct format.
Benefit
The benefit to the end user is a ready made application server software with software maintenance. The software provides a fast development time without the uncertainty of the development of new software for the purpose. The quality of the application software improves with less cost and the software can be distributed to all of the application server users. It is possible to integrate multiple databases into an information system since the schema may consist of attributes in multiple databases. The attribute dependencies can be given new meanings based on the needs of the information system and the dependencies of the attributes may be more informative than the ones in the relational database schema. The application is able to use international character codes and texts.
Architecture
The application is a stateless program with an external session database. Every request gets a fresh copy of the processes and after each request, all of them exit. The memory between the requests or the session information is kept in a separate session database or databases. The actual database connections to the background relational databases can be added by loading and using external database modules. The concept has the following components needed for different purposes.
Program interconnections
Program interfaces and possible architecture component interconnections are listed in the image "Possible interconnections". An HTTPS -server may be used in front of the application server to serve the not changing content.
Session database
Session databases can be added to be used in a fault tolerant configuration. Each is able to distribute the load to other session databases. One session database server may fail if more than one session database are added. If more than two servers are added, the servers are able to share the memory and connections between them. The program logic is in the client software inside the RVP application. More information on how to add session databases can be found from "Configuration Options" section and more specifically from configuration settings of setting "SessionDB".
Session database saves the attribute name-value pairs for the session lifetime. The attributes are usually used in identifying purposes and in giving permissions for the user to certain transactions. The session database can be said to be a role database or an identity attribute database, the correct name is a session database. For example the user login is a transaction where the user is given permission by identifying the user. As a result in the example, an attribute is saved in the session database (setting "SetRoles" in the "Transaction" "Roles") object. To identify the browser, the application server uses either cookies or session attributes for the session identification. The application server may use any attribute, for example an email address to identify the user.
Databases
Database can be any database, for example a file or a relational database. Access to databases can be added by adding modules. The transactions to the database have to be atomic as in the relational model. For this reason, the background database should be a relational database. Otherwice especially in reporting, the database can be any database, for example a file and the attributes reported may be used without the schema. If the atomicity of the transaction can be provided, the database can be any database. A database module may be used in any purposes. This is a possibility to use to develop many different types of applications.
HTTP-Server
The RVP may connect through an HTTPS or HTTP-server acting as a proxy or by its own as an HTTP server. The HTTPS-server may be used to decrypt and encrypt SSL. The traffic from RVP is ordinary HTTP added by cookies with optionally secure enabled. Using a proxy to forward traffic is modular. Using proxy, the HTTP-server runs in parallel with the RVP to use the server resources efficiently. The HTTP-server serves static files like images and html-pages in parallel with the RVP application server. The proxying HTTP-server reduces the load from the application server when the application functionality is not needed. Additionally the HTTP-server decodes the secure sockets layer. The HTTP-servers are usually state of the art software with useful settings sometimes needed and it is not reasonable to add all of them in the application server. New features like for example HTTP2 or the HTTP version 2 may be used with this configuration. Instructions to add a proxy function can be found from the section "Attaching to HTTP-Server".
Program processes
During each request, program execution divides into three processes. The processes are isolated from each other and run in their own reserved memory. This adds security in the program. The database connection is separated from the consistency and the integrity check in its own memory. After each request, the processes exit emptying everything the previous request had left.
CBI
CBI-process is responsible in checking the consistency and integrity of the input attributes. CBI-process reads the transaction definitions, compares it to the attribute definitions and determines the combination of attributes to exist before the transaction is possible. It verifies the consistency of the attributes reading the session database attributes (marked with & -sign in the transaction definitions) and the input attributes (marked with $-sign or #-sign). It is possible that it is trusted that some attributes are in the database, still partisipating in the referential integrity check (marked with @ -sign). CBI validates the value of the attribute content with the attribute definitions type declarations. Transactions can be repeated, SQL-statements can be repeated and multiple databases can be used in multiple calls in a call to an URL.
DB
DB-process uses the actual database module to connect to the database and to implement the actual transactions, writing the results in name-value pairs to the reporting process, committing the transaction in the database and closing the connection. DB may use multiple modules to connect to different databases. Database can be any database, an authentication function, an email sending application, a file or a relational database.
TAG
TAG-process reports the output in name-value pairs get from the DB-process. TAG uses tag names inside the output files to recognise the places where it outputs the data. CBI can signal DB and TAG to start writing the error page instead and all CBI, DB and TAG are able to start the system error page if something went wrong. Typical error situations are a missing program component, a database for example. A descriptive error message helps to reason the cause and to correct the program behaviour. More information can be found from the sections "Reporting" and from "Reporting Options".
Application Development
Output Files
The output is printed in the output files. The output files have text strings (starting with text <%TAG ) in them to recognize the places where the attributes values are printed. There are three types of output files: the files used in normal output, the files used in error conditions, when the consistency and integrity check has failed, and the files used to print system error messages. Every transaction has both of these output definitions in parameters "Page" and "Error page".
The error pages may be the same pages where the output is printed if this is reasonable in the user interface. The error pages are used to print the error messages from consistency and integrity tests. The messages should inform how the attribute values should be given for the application.
System error pages are special conditions reserved for the application software error messages. These are the standard HTTP error messages and the internal reason code messages of the RVP. Every module developed to the RVP may add more error codes and error definitions and they may use the already defined error messages. The error codes and the corresponding error messages are configured in run/etc/error.conf. New error codes should be defined above value 700. The location of the system error files are by default run/sys/error.html and the syntax error snippet is defined in run/sys/syntax_error.html.
Message content encoding may be any text file encoding in the RVP character encoding: HTML, XML, JSON or any. The most used in WWW-applications is HTML in its various forms.
MIME Types and HTML Forms
RVP understands both HTTP POST and GET queries. The capability to use query part of the URL may have to be configured in the "UseQuery" parameter in the run/etc/rvp.conf. The query key-value -pair has to end in & character in the end of the query. It is recommended to use method POST because the content can be in internationalized encoding and the length of the message content may be longer. Web browsers have different maximum length values for the URL ranging usually from 2048 to 4000. Best MIME type to use is application/x-www-form-urlencoded in both GET and POST queries. In HTML FORM:s, this is the enctype parameter. enctype sets the HTTP Content-Type -header when POST is used. text/plain may be used if the values do not have linefeed in them and it is not anymore recommended. If the urlencoding is needed using GET, the "URLEncodedValues" setting should be set in the configuration. Character sets are chosen using accept-charset.
JSON
JSON format is used more and more today to exchange data. To use JSON output, the JSON file can be formed with the <%TAG -output or a database module can be used to return the JSON file allways in an attribute, for example in attribute named ‘JSON’ text. The JSON output can be used by setting the correct MIME-type application/json, setting the UFT8-encoding and by choosing the output format.
Encodings
Available encodings are one-byte encodings, two- and four byte encodings and all of the Unicode encodings UCS-2 and UCS-4 and the transfer formats: UTF-8, UTF-16LE, UTF-16BE, UTF-32LE and UTF32BE. These have to be set in the encoding settings and the setting is used in every transaction. Every external interface may have different encoding. All of the input has to be in the same encoding. One byte encodings include 7-bit US-ASCII and all of the one byte encodings, at least ISO-8859-1 and ISO-8859-15, the Scandinavian one byte character set. Recommended default is UTF-8. Unicode character sets contain all of the characters in the world.
Attribute declarations
Automated schema
The application server forms an attribute schema based on individual attribute declarations in each startup. The schema is similar to the relational database management systems schema in table format. The difference is the attributes are declared individually. The relational database schema can be used in adding the software attributes. The primary-key attributes are the determinants or the multivalued determinants. More determinants are found in the same tables. The attribute declarations are otherwice similar to the declarations in the SQL. The convenient error messages are attached in every integrity type of an attribute to help the user to type the attribute values in correct format.
In some extent it is possible to use the same already configred run/etc/attributes.conf -attribute schema or attribute catalog in multiple applications. In using multiple databases, the attribute names still have to identify the attributes.
Relational integrity
Relational integrity is defined with the following four rules. Different sources add other rules, these are used here. In a transaction to a relational database, the attributes modified are allways the ones in the relational table or a relation where the attributes are. These transactions are called atomic. The RVP software checks the following four different types.
Entity integrity
The relational database schema defines tables. Every table has to have at least one key attribute. One of the so-called superkey attributes are chosen as a primary key. The key values may not be null. This is the entity integrity. If an attribute is declared as a primary key with "PrimaryKey", the attribute may not be null in the transactions. Every determinant is automatically a key.
Domain integrity
Usually a key value forms a domain of an attribute. In this case the same named attribute is in two or more tables forming a domain in between the tables. In principle, the value of an attribute should allways be one of those values of a primary key attribute. Programmer has to ensure this, the RVP can not ensure the domain integrity and so the domain integrity should be designed in the application by the programmer. Relational database software is usually able to check domain integrity denying the transaction if the attribute value is not from the domain.
Sometimes the domain integrity can be ensured by choosing a correct identifying attribute in the transaction. For example if the identifying attribute identifies the user, the identifying attribute usually identifies the transaction attributes including the domain attribute as well. If not, the application programmer should provide the correct values of the domain attribute from the primary key values with some other attribute.
|
|
The RVP does not check the domain integrity. The domain integrity is ensured by selecting the correct values in the WHERE -clause of the SQL-statement. |
Null integrity
Null values are usually used as a placeholders to add data later. Different attributes may be set as "NotNull" true. If "NotNull" is used, the attribute is always checked to exist and the integrity of the attribute value is always checked.
Referential integrity
Determinants
Foreign key is an attribute in more than one relational table. According to the referential integrity, the foreign key must be null or it must match a primary key of another table. Referential path is the path of every determinant of the attribute and the determinants of the determinants. Referential integrity is the existence of every determinant in the referential path. The RVP ensures the referential integrity by using "Determinant" and "MultivaluedTo" -definitions.
In short, all of the primary key values in a relational table are determinants of all of the attributes in the relational table. The relational tables usually have more key attributes, superkey attributes or other primary key candidates. These should all be determinants of other attributes in the table. Application functionality may reveal more determinants to be configured in the ‘run/etc/attributes.conf’.
Multiple declarations of "Determinant" should be added if more than one determinant is needed.
Determinants of multivalued attributes
Multivalued determinants are slightly different. Multivalued dependencies should allways occur in separate relational tables if the relational database schema is a design conforming to the normal forms. Usually the multivalued dependencies are between the domain attributes and the attributes in the table where the the domain attribute is not a key of the domain. In the DK/NF (domain-key normal form), all of the multivalued dependencies are removed from the tables. The integrity check of the referential integrity has additional rules for the multivalued dependencies and multivalued dependencies may be used in the application.
Multiple declarations of "MultivaluedTo" should be added if more than one determinant to a multivalued attribute is needed.
Configuration file format
The configuration files are JSON files started with an object sign ('{' - curly brace). The usual location where the files are found is the directory run/etc/. Unlike in JSON depending on the interpretation, the special control characters are allowed, for example the ones below Unicode value 0x001F. All application reserved special characters have to be bypassed, for example the backslash '\'. This is important to know with regular expressions.
‘run/etc/attributes.conf’
The attribute definitions in the previous section lists the most important declarations of the attribute declaration file. run/etc/attributes.conf is a JSON file. Attribute names are added as a list. Every attribute is an object containing the prementioned relational integrity parameters with the following objects: "Type" and "Error".
Application reserved special characters are the characters used to delimit names and values and the bypass character. The texts in the configuration files are usually in the encoding of the output. The most used is the HTML. The special characters of these have to be bypassed in the texts. For example in the regular expression of the email -type in the previous example, all of the bypass characters have been bypassed with another bypass character.
Since in most cases the editing of the files has to be done by hand, the normal line feed characters may be used instead of using only the JSON special characters. In checking the syntax of the file, ./run/bin/rctrl may be used to remove these special characters before checking the syntax of the file.
|
|
The first bypass is removed in parsing the JSON and the second bypass bypasses the program special characters. It is reasonable to use percent encoding in URL-strings to avoid extra bypassing. The & -character is used in marking the attributes from session database and it is at the same time a special character in the URL query -text separating the key-value pairs in the URL. |
Type declarations
It is important to use a typesystem to check the form of the type. For example in using SQL attributes, a check if the attribute value has too many parentheses may prevent unwanted use of the database by the use of unwanted nested SQL-statemens. The ‘attributes.conf’ error messages can be used to give instructions to the user to give the attribute value in the correct form helping the user in using the application.
The most important type declaration is the maximum length "MaxLength" of the attribute value.
The form of the value of the attribute type can be checked with a typechecking module. In this case the "InBuilt" declaration can be used. If attribute name is different than the attribute name in the module, the parameter "InBuiltName" may be used to use the needed name from the module data types. The corresponding module declarations in ‘run/etc/rvp.conf’ are found from section run/etc/rvp.conf.
The form of the value of attribute type can be checked with a regular expression. Parameter "RegExp" is used to give a regular expression. The expression may be matched maximum of "MatchCount" times or if it is not defined, once. Regular expressions are from Perl compatible PCRE2 regular expressions. They are used in 32-bit mode. Matching buffer is used again with an overlap if a too large value is encountered. JSON special characters ‘"’ and ‘\’ have to be bypassed in the regular expression text.
An another attributes type declaration may be used with parameter "BaseType" and additional type checking can be added. If a type declaration is defined, the value has to be in the form of all of the declarations. At least one attribute type declaration is needed. If the "BaseType" -attribute is not found from the ‘run/etc/attributes.conf’, the attribute is searched from ‘run/etc/types.conf’. Attribute type checking may be chained this way. Each additional typecheck increases the time used in typechecking. This forms a typesystem of the attribute values.
Error definitions
Each time a consistency check notes an error or if the attribute values form is in error, an error message is printed to output to be preinted in an error page. Error pages are configured in ‘run/etc/transaction.conf’. The printed error attribute names have a dot notation. Attribute name in error are appended by dot and a text "error". This text is appended by a dot and the error type definition. The error type definitions are: entity, dependencies, role and type. "Role" should describe the origin of the attribute to be used in the session database.
For example if a role dependency of an attribute email was missing when a transaction was tried, a text:
would be printed to the output to the reporting process named TAG. This attribute can be used in the error page to inform the user that the required attributes were not available for the transaction:
The configured error page should have all of the possible error messages configured in the correct places of the file and if the error message occurs in the output, its message content from the typesystem is printed in the error page output.
Error definitions are configured in every attribute object in an object named "Error". The values of the error attributes are read from these four different error type values:
|
|
The purpose of the error messages is to help the user to give the values in the correct form and to help in using the application. |
Transactions
Request URL:s are associated with transaction definitions in run/etc/transactions.conf. the configuration file is in JSON format and every URL defines a new JSON object as a list. Every object has "Page" and "Error page" definitions, "Roles" definitions and the actual "Transaction" definitions. Page definitions are file paths from the program root, the run directory.
Transaction
Transaction object is the connection to the database or other module producing the output of the transaction. The parameter to the module may be a "SQL-Statement" or a "Parameter". The difference between these two is in replacing the empty value. In the SQL, empty values are replaced with a text NULL. In these texts, the input parameters are marked with a dollar sign $. The RVP replaces these variable names with the attribute values.
"Module" parameter chooses the needed module file installed in run/lib -directory and configured in the run/etc/rvp.conf -file. "Modifies" setting is important. If it is set to false, the referential integrity is not checked. in the SQL SELECT statements, the referential integrity is not necessary. It is easier to read from the database than it is to write.
Other optional parameters are the "Repeat" and "Returns". If "Repeat" is set to true, the application will repeat the transaction to every attribute set it finds from the input. Care should be taken to not to use this function in anything where it may cause too many rows or unintended data. "Returns" attribute declares an error attribute name the developer may use to deliver the database module error message to the user interface.
The previous example would select the user email to be set in the session database as an authentication function.
|
|
It is advisable to leave the Modifies to true to be able to check the consistency of the attributes. For the referential integrity, it is important to check all of the determinants of the attributes in the transaction. In the relational model, all of the determinants of the attribute must exist before the transaction may be attempted. |
The Modifies attribute value false switches the referential integrity check to off. The SELECT-statements do not necessarily need the referential integrity check. The developer should take care to SELECT only those attributes needed for the transaction using the session database attributes and the input attributes.
"On success"
The database module should always return a success or a fail return value. With the return value of a failed SQL-statement execution function, a descriptive text is returned to the output to be printed in the results page in an attribute named with "Returns" attribute. On successful operation, the RVP executes the SQL-statement or other parameter in the "On success" parameter. This is usually a COMMIT -statement. Every succeeded transaction executes the "On success" even if the next transactions would fail. Same attributes are used. A SELECT can be used here as well to return output from the database module if needed.
Using attributes in transactions
The transaction can be repeated if the setting "Repeat" is set to value true. CBI will repeat as many times it finds the next names of the transaction from the input stream and the DB will repeat the transaction as many times the attributes are found.
The attributes are grouped in every repeat. Allways the next same named attribute is get from the inputs. If "Modifies" is set to true, unique input values are used. In this case allways the first occurrence of the attribute is used. Inside a repeat loop, the attributes are grouped and inside the group number and the repetition, the attributes are unique.
Every SELECT -statement has to be in its own "SQL-Statement" -block. INSERT and UPDATE -statements may be in the same "SQL-Statement". It is as well possible to use an own or multiple "Transaction" -blocks. In this case all of the database and the connection information has to be written again and each block has its own "On Success" -block for a COMMIT -statement. It is possible to use as many databases with as many database connections as is necessary.
If these characters are needed in the "SQL-Statement" text, these can be bypassed with a double bypass character: `\\' . Since the "Parameter" text is not in SQL syntax, single bypass is needed `\' .
Attribute search order
Each "SQL-statement" and "Parameter" -block starts an own attribute group. Inside the group, only one similar attribute name is found. This is the same as in the Grouping search in the reporting options. Inside the group the behaviour is similar as in the Unique search ordering.
The attribute prefix sets the attribute as not found after the search and sets the same attribute as reusable in the next searches. The attributes are searched as if they were Unique across all of the "SQL-Statement" and "Parameter" -blocks in the URL.
Roles
Session attributes are set with "SetsRoles" and used with "ReqRoles". To use the session attributes in an SQL-Statement, a & -sign is used. Session attributes are used as unique attributes, they may occur only once and when they are used, the attribute value is always the same value.
|
"SetsRoles"
|
List of attributes to get from the database module output. These are saved in the session database. The type integrity of the attribute value is checked from the module output. If at least one attribute is found from the database module output, only this one role is set and normal operation continues. If no attributes matched, an error page starts. |
|
"ReqRoles"
|
The list of attributes required before the transaction is possible. These are the identifying attributes found from the session database. If set to boolean value false, any use of the URL is allowed. For example in the login function. |
|
"RemoveRoles"
|
RemoveRoles lists the roles to remove after the transaction. |
|
"Logout"
|
If Logout is set as true, the session identifier information is removed from the session database and the web browser. |
|
|
It is important to define all of the correct role -attributes using ReqRoles in each transaction. If the roles are not added in the requirements of the transaction, anyone can use the transaction as they wish. The roles or the identifying attributes are important. |
Login
The login function can be set with "ReqRoles" boolean value false. In this case "SetsRoles" should contain all of the attributes needed from the login function output on a successfull operation. Program saves theses attributes in the session database and they can be used in every transaction where they are configured in "ReqRoles" -setting as a JSON text object.
Identity of attributes and roles
The consistency of the transaction attributes and the integrity of each attribute is checked from each attribute of the SQL-Statement marked with a letter $ and &. The attribute is fetch from the session database, after this from the message content and after this if set, from the query -part of the URL.
Session database should contain only attributes used to identify other attributes. These name-value pairs may occur only once. The names have to be unique. If in the "Transaction" -blocks "SQL-Statement" or in the "Parameter" -block if one of these parameters are needed, a & -sign is put in front of the attribute name. If in the parameter block it is assumed that the attribute is found from the database already and its value is not needed here, the attribute is prefixed with a @ -sign. If the attribute value is read from the input from the message content or from the URL query -part, it is prefixed with a dollar sign $.
The names of the attributes in the session database, the role attributes or the identifying attributes should be renamed to be identified as session database names. Otherwice the roles and the identifying attributes may mix with the other attributes with the same name. This way the integrity check is simple. The consistency check tries to fetch the attribute first from the session database attributes and after this from the inputs, from the message content and from the query part if this is set. It is possible to use SQL AS in the "SQL-Statement" to rename the attribute to another to keep the attributes unique.
Using ReqRoles and SessionDeterminant -attributes
The "ReqRoles" -setting in the transaction block is used to restrict the use of the transaction to only those users who have the "ReqRoles" -attributes set in the session database. The application won’t do the transaction if this attribute or a name-value -pair is missing from the session database. A system error page will result from an attempt to use these URL:s with these transactions without the attribute in the session database.
|
|
If the attribute requirement is the same in all of the transactions, the SessionDeterminant in ‘run/etc/rvp.conf’ can be used to attach the attribute requirement in every transaction of the application. |
Reporting
The program output is formed from the database module output. Please see the reference to see all of the available options. The database module may report a delimiter separated list or attribute-value pairs. It is possible to add text before and after the attributes containing attributes. The attributes can be repeated to form tables or arrays, for example HTML tables or similar. For example printing a CSV-file is simple using a file -module.
System Administration
Session identifier
The application uses session identifier to recognize the application session. This is usual with application servers. The session identifier is used as a key to store the session data in the session database. Therefore the identifier should be kept safe. When it is possible, HTTPS should be used. Session identifier attribute name is set in "SessionID" in rvp.conf if it needs to be changed. It is normal to use the cookies to recognize the browser the user is using and sometimes a user concent is needed to approve this to inform if the cookie is used to track the user.
The attribute may be delivered between the web browser or other client in session cookies or in attribute values. Using attribute values is actually difficult because the user may accidentally stray away from the returned www-pages and the session identifier would be lost. It is of course possible to use JavaScript or other functionality to bypass this and save the session identifier in the WWW-browser. If attribute values are used, the attribute "InputSessionID" should be set to true.
Attaching to HTTP-Server
Many WWW-hotels have a www-forward functionality to be used. They may be available without SSL support. It is still reasonable to use a proxy in front of the RVP. All of the reverse-proxy servers will work with the RVP and all of the HTTP-servers with a proxy functionality. If SSL is needed, the proxy should be able to decode SSL.
An example using Apache httpd
Simple configuration can be made using Apache httpd modules mod_proxy and mod_proxy_http, a simple configuration using only one RVP -server in Apache httpd.conf.
The requests to URL /live are sent to the RVP in address my.domain.com.
Similar functionality is get with different load balancing features using Apache httpd modules mod_proxy, mod_proxy_http and mod_proxy_balancer. An example of using multiple RVP-servers with different balancer modules, here, the "byrequests" is used.
Directive Proxypass tells the path of the URL from where the requests are send to the RVP in port 8443. In the example this is /live. Timeout has to be set to a bigger value than the biggest timeout in the rvp.conf . The default value in Apache 2.4 is 60 seconds (as of 13:th May 2017).
Configuration Options
Startup parameters
Reference
Program directories
|
bin
|
Program binary files. |
|
lib
|
Program shared libraries. |
|
module
|
External modules. |
|
etc
|
Configuration files. |
|
log
|
Log file directory if another filename is not given in rvp.conf. |
|
sys
|
System error files if another file paths are not given in rvp.conf. |
|
var
|
Variable files used by the program. |
|
www
|
Output files and other content to use in output. |
Additionally the /tmp -directory has to be writable.
run/etc/rvp.conf
|
OutputContentType
|
The MIME type of the output. For example text/html. The file format of the files in run/www. If left empty, default value is text/html. |
||||||||||||||||
|
UseInsecureHTTP
|
If set to true, the Secure text is not used in the cookies. |
||||||||||||||||
|
URLEncodedValues
|
To use URLEncoding to read the attribute values. |
||||||||||||||||
|
Domain
|
Domain to be used in the cookie value. |
||||||||||||||||
|
URLPath
|
Path to be used in the cookie value (the same path used to use the RVP) |
||||||||||||||||
|
SessionID
|
Name of the session identifier attribute in the session cookie. |
||||||||||||||||
|
SessionDeterminant
|
Name of the role attribute used in every URL. The same if individual ReqRoles -attributes are defined. |
||||||||||||||||
|
InputSessionID
|
If set to true, session identifier is get from attribute named in SessionID from the POST or GET content and the cookie is not set. |
||||||||||||||||
|
InstanceID
|
The distinguished name of the server (a unique server name). This attribute is needed to make sure the session key is unique. |
||||||||||||||||
|
SiteID
|
Identifier of the domainname and the path. For example the company name, the application name or similar unique name to be used in every server of the site. |
||||||||||||||||
|
License
|
A license key in a hexadecimal text string. The length is normally 64 characters. |
||||||||||||||||
|
ReadEncoding
|
Default encoding used in reading from the input (for example UTF-8). |
||||||||||||||||
|
WriteEncoding
|
Default encoding used in writing (for example UTF-8). |
||||||||||||||||
|
Error Page
|
System error page. Default is run/sys/error.html. |
||||||||||||||||
|
ErrorDefinitions
|
Name of the file containing system error codes and error messages. Default is run/etc/error.conf. |
||||||||||||||||
|
TransactionDefinitions
|
A transaction configuration file. Default is run/etc/transactions.conf. |
||||||||||||||||
|
AttributeDefinitions
|
An attribute configuration file. Default is run/etc/attributes.conf. |
||||||||||||||||
|
DerivationTree
|
Directory and file name to use to write a temporary file during the program startup. Timestamp of the file is compared to the timestamp of run/etc/attributes.conf. |
||||||||||||||||
|
Logfile
|
Log file. Default is run/log/rvp.log. |
||||||||||||||||
|
UseQuery
|
If set to true attributes are searched from the query part of the URL additional to the POST input. |
||||||||||||||||
|
EncodingErrors
|
If set to system error, every character encoding error cause a system error page. If set to conditional, first occurred character encoding error is given if the attribute integrity check would otherwice be successful. Default is to allow all encoding errors. The typecheck should test the values. |
||||||||||||||||
|
SQLEmpty
|
Text used to replace missing attribute in "SQL-Statement" -text. The default is SQL NULL. Note that empty values are not replaced. |
||||||||||||||||
|
Empty
|
Text used to replace missing attribute in "Parameter" -text. |
||||||||||||||||
|
SessionTimeout
|
Number of session timeout in minutes. The time in the cookie is this value in GMT+0. |
||||||||||||||||
|
RequestTimeout
|
Time in seconds to wait for the transaction to complete before the connection is shut down and the process is quit. |
||||||||||||||||
|
ServerAddress
|
The IP address and port number in the form <IP>:<Port>. A hostname or IP may be used here. |
||||||||||||||||
|
RedundantServers
|
Number of session databases able to replace each other. Number one means the data is written in one session database. Default value is two. |
||||||||||||||||
|
LogLevel
|
Number between 1 and 8. Number 1 logs only the necessary output, number 8 is used in problem resolution. Number 6 is enought to log the client connections. |
||||||||||||||||
|
CAFile
|
Certificate authority file path to deliver to the database modules. Default is /etc/ssl/cert.pem. |
||||||||||||||||
|
SessionDB
|
A JSON object containing the names of the session databases as JSON objects. The names are case sensitive. Note that the session database IP-address may not be the same as the server IP-address connecting. The settings are:
|
||||||||||||||||
|
DBModules
|
Database modules are declared in a JSON object similarly. Module name starts another JSON object containing:
|
||||||||||||||||
|
TypeModules
|
Type modules are likewise defined in a JSON object with the case sensitive type module names starting a new JSON object with the following settings:
|
An example run/etc/rvp.conf can be found from the modules section.
run/etc/attributes.conf
The run/etc/attributes.conf is a JSON file with a list of names. The name is the query part of the URL or the filename of the configured path. See examples.
|
<attribute>
|
The name of the attribute is the JSON attribute object. Attribute -objects form a list in the attributes.conf -file.
|
run/etc/types.conf
The form of the file run/etc/types.conf is the same as is the form of the run/etc/attributes.conf -file. The types.conf is used as an additional file to use the "BaseType" settings and to clarify the applications own attribute definitions. The types.conf could be used again in an another application.
run/etc/transactions.conf
The run/etc/transactions.conf is a JSON file with a list of names. The name is the query part of the URL of the configured path or the filename.
|
<name>
|
The name is the query part of the URL of the configured path or the file name.
|
run/etc/errors.conf
run/etc/errors.conf is a JSON file containing two kind of errors: The standard HTTP errors and the reason codes used internally in the RVP. The codes are associated with a reason text string giving the user more information on the occurred error. These text are printed in the system error page output.
|
"Http"
|
The JSON object contains a list of three digit HTTP error codes as a JSON string. |
||||||||||
|
"Server"
|
The JSON object contains a list of three digit error codes used in the RVP.
|
Log file
Log options are ‘Logfile’ and ‘LogLevel’. These are in ‘rvp.conf’. The level is a number between numbers 0 and 8. Number 8 logs the most. Usually required connecting client IP, port and connection time is printed with log level 6. The numbers are similar as in syslog:
|
0
|
Only the necessary output. |
|
1
|
Log only emergency warnings. |
|
2
|
Log alerts messages. |
|
3
|
Log critical messages. |
|
4
|
Log errors messages. |
|
5
|
Log warning messages. |
|
6
|
Log notifications. |
|
7
|
Log more information. |
|
8
|
Debug logging used in problem resolution. |
Special characters and bypassing
The configuration files are in JSON format. The JSON special characters ", \ and control characters have to be bypassed in JSON text as in the previous chart of special characters. Because the same bypass character has been used in the application special characters: $, &, # and @, double bypassing: ‘\\’ is needed in front of these special characters. JSON does not allow not bypassed special characters.
Checking the syntax
The developer should ensure the syntax of the files. It is advisable to use a validator to check if the configuration files are in JSON format.
Reporting Options
Output from the application server may be done replacing a special string in the www-pages with attribute values. The string is called a tag and its form is:
The options texts are used between the texts <%TAG and $attribute. Special text can be used to denote usual special characters similarly as in JSON: \n (line feed), \t (tabulator), \r (carriage return), \b (backspace), \f (form feed), \\\ (backslash), \/ (solidus), \" (quotes) and any Unicode character in a form \u followed by four hexadecimal digits of the code. The group tag forms a repeatable block of text, attribute and other group tags to repeat.
Attribute search order
The attributes reported are searched from the database module output. Allways the next same named attributes are searched and printed. If the attribuet or the next one is not anymore found, the attribute is not found. There are possibilities to change the search order: UNIQUE sets the attribute as unique. With this setting the same attribute is used again and again. The first occurrence of the attribute is used, either from the last group or from all of the attributes. Every REPEAT loop changes the group of the attributes in a loop. In a loop, the same attributes are used again and again as with UNIQUE. These attributes are inside a group and the group is changed with every loop. Every loop increases the group number and new attributes are searched and used within a group.
|
Next
|
Search finds allways the next attribute, if it was the same named as the previous or if it wasn’t. This is the default in TAG. |
|
Unique
|
Search finds allways the first and the same same named attribute. This is the UNIQUE setting. |
|
Grouping
|
Search finds the next groups attributes as in repeats. Inside the group the attributes are UNIQUE. The grouping is used automatically inside the TAG. |
Option texts to delimit output
Options in attribute tag.
ST
Text defined in ST is printed before the attribute value defined with a dollar sign.
END
Text defined in END is printed after the attribute value defined with a dollar sign.
MIDDLE
Text defined in MIDDLE is printed in the middle of the repeated attribute values defined with a dollar sign.
EMPTY
If attribute value did not exist in the result set, this text is printed instead. Empty value is printed as a zero length text without start and end texts.
MISSING
If attribute did not exist in the result set, this text is printed instead. Empty value is printed as a zero length text without start and end texts.
Nested output
It is possible to put inner <%TAG:s inside the options texts in the <%TAG. The text is replaced with the <%TAG value from the database module output.
Inside the content of the options texts the special characters have to be bypassed to deliver them to the next tag. In the above example, if the inner tags had options texts, the quotes of the texts should be bypassed. If a syntax error is encountered, the RVP usually prints the system error page with an ascii arrow pointing to the location where the error occurred.
Repetitions
REPEAT
A number in the value describes how many times the <%TAG is repeated. All of the nested attribute values inside the <%TAG are fetched again from the database module output. The attributes are grouped in every repetition. If an attribute has to be printed twice or more in a repetition, the printed attribute is the value of the same attribute. The next repeat changes the group to the next group and the next attribute is fetch in the next repetition. If the number is -1, the repeating is endless.
CONSTANTREPEAT
The constant repeat repeats the same text as many times as is the number in the text value. Attribute values are the same and new attribute values besides the <%TAG attribute itself are not fetch.
MIN
Field count from which to start printing.
MAX
Field count to stop the printing.
CHARCOUNT
Maximum number of characters to print.
Separators and texts of tabulated output
Depending on the database module, the attribute value may contain a tabulated stream of delimiter separated values. The <%TAG is capable of parsing this text if separators are single characters delimiting values in an output stream.
The order of the separators are as follows: <file <group <record <unit> > > >, the unit is on top of the others and the first separator is the file separator.
Separators
US
Unit separator.
GS
Group separator.
RS
Record separator.
FS
File separator.
Separator texts
UNITST
Text defined in UNINTST is printed before the attribute value of the tabulated output of US -delimiter separated values.
UNITEND
Text defined in UNINTEND is printed after the attribute value of the tabulated output of US -delimiter separated values.
RECORDST
Text defined in RECORDST is printed before the attribute value of the tabulated output of RS -delimiter separated values.
RECORDEND
Text defined in RECORDEND is printed after the attribute value of the tabulated output of RS -delimiter separated values.
GROUPST
Text defined in GROUPST is printed before the attribute value of the tabulated output of GS -delimiter separated values.
GROUPEND
Text defined in GROUPEND is printed after the attribute value of the tabulated output of GS -delimiter separated values.
FILEST
Text defined in FILEST is printed before the attribute value of the tabulated output of FS -delimiter separated values.
FILEEND
Text defined in FILEEND is printed after the attribute value of the tabulated output of FS -delimiter separated values.
Replacing separators with text
UNITMARK
Replaces the character in the output defined in the US separator field with the given text.
RECORDMARK
Replaces the character in the output defined in the RS separator field with the given text.
GROUPMARK
Replaces the character in the output defined in the GS separator field with the given text.
FILEMARK
Replaces the character in the output defined in the FS separator field with the given text.
Other options
UNIQUE
UNIQUE can be used with a text true or without a text. This setting sets the attribute as unique. Each time the attribute is used, it is allways the first attribute found from the stream.
BYPASS
Bypass characters are used to bypass a control character in the database module output. The BYPASS attribute can be used to replace this one character to another.
LOOSE
If set to ‘1’, separator does not start the underlying groups.
REMOVESPHT
Removes extra space and tab characters.
LWSPFOLDING
Read the output file unfolding the text as in email header texts.
Grouping attributes using group tags
To add text before, after and between group of attributes, paired tag <%GROUP> <%/GROUP> and <%GROUP $attribute> <%/GROUP> can be used. The first pair without the attribute definition repeats as many times attributes are found. This is attributegroup. The second can be used to repeat as many times as the domain attribute value changes in the group. This is domaingroup. In domaingroup the repetition stops when the domaingroup attribute value changes. In selecting from two relational tables, the domainattribute in both tables creates repetition. The domainattribute value has to be ordered. The change of domain is recognized from the changed domain attribute value.
It is possible to report JSON-files and other tree structures from the relational database tables using the domain attribute. In the example the transactions.conf -file is printed using attribute tags and group tags. The attributegroup may be used before the domaingroup to endlessly report as many times there are attributes.
Option texts to delimit output
Options in group tag.
MIDDLE
Text defined in MIDDLE is printed in the middle of the repeated groups.
Special characters and bypassing
The database module output may contain control characters with a special meaning. A bypass character \ in the database module output causes the software to bypass the control character functionality and to remove the bypass character. A bypass character in the database module output is printed with two bypass characters.
Modules
Modules are added in the RVP wide configuration in run/etc/rvp.conf. The modules are read and initialized during the program startup. Adding a module requires a restart of the program. See the possible signals RVP uses for other possibilities. A module is external program code run inside the RVP. It is possible that during the startup the RVP will stop after trying to load the symbols of one of these modules. In the log file the loading of the symbols is the last visible line if the loglevel is increased to the debugging level number 8.
Two types of modules exist. Typemodules check the form of its input variables type. Database modules connect to the database, executes a transaction and closes the connection. After a successful execution, a commit execution will follow if the transaction is set to modify the data. Modules are added in the run/etc/rvp.conf.
In using database modules in authentication, it is sometimes needed to deliver all of the request message content to the module. If "DirectInput" is set to true, the consistency check of key-value pairs is not done and all of the URL query and message content is delivered to the database module as an input. This is useful in developing authentication modules using standard protocols, for example OpenID or SAML.
Most default values of the settings may be used instead of writing all of them in the rvp.conf. Module names are case sensitive and usually only ASCII letters are possible because of the connection with programming external modules. Almost all of the texts are in UCS four byte character format.
Example ‘rvp.conf’
In the following is an example rvp.conf loading modules, a PostgreSQL database module postgres, module email sending emails with SASL, a module file reading output from a file and a typemodule builtintypes. Module name chart prints tabulated text from the PostgreSQL module with the chart functions. The function names are changed with "FunctionPrefix" and the given module name can be any.
Example modules
Included in the RVP distribution, some ready made modules are available. For example the following modules to use database connection to PostgreSQL 9.6 database, SASL authenticated smtp emails module and a module printing just an output file. Configuration example to add these modules in rvp.conf was in the previous section.
Module file.so prints the files declared in the transaction in an attribute. The file is output to the attribute value. The attribute name is postfixed with an order number. For example the following files in the example would be printed in attributes FILE1, FILE2 and FILE3.
cIf the file is needed to be output as itself, a file containing only the <%TAG -text can be used in outputting. Be aware that the empty white space characters are going to be in the output. In these cases the MIME-type in the HTTP headers set in the "Content-type" -header is the same the application is using. This can be changed with the "MIME-Type" setting of the URL. A comma separated file for example can be printed with the <%TAG as in the example.
Email is usually sent with SASL authentication. This module connects to the SMTP server, authenticates and prints a standard MIME-type format of an email to the SMTP server. Note that the header attributes start from the first character of the line.
A module tag.so forms a file using the RVP output format. The module reads a first parameter file and writes attribute data in it to record the result in the second parameter file. In this example, a configuration file. The file format of the first parameter is the same as in the "Reporting Options" section.
Developing database modules
The database modules are external programs to be used by the RVP. The RVP finds the implemented functions from the library file using the symbols of the functions. Other functions can be declared as static for example to not to mix them with the symbols used. For example nm -tool can be used to check if the needed symbol is available. To export the symbols, a compiler flag -export-symbols or similar is needed. Some operating systems allow additionally initialization functions to be used in source files. These functions are called at the time when the library is loaded. In the example modules these init functions are not used.
The function interfaces to be implemented are declared in an include file cb_db_module.h.
The database module name is used as a prefix in searching the functions from the library file. If attribute "FunctionPrefix" is used, this prefix is used instead.
Connecting to the database usually results an object in reserved memory. This object can be carried in structure db_conn_param in pointer *dbconn. The pointer to the memory is copied to structure transaction_params pointer *dbconn. Allways make sure the pointers are either NULL or carry allocated memory. Other options are explained in the include files and in other external sources. Usually every text is encoded in four byte UCS-4 representation. The module program should allow some other parameters to be NULL as well. For example the *param and *queryparam parameters are set to NULL if DirectInput was used.
The "SQL-Statement" is a text in the chosen encoding in the parameters *sqlst and *utfsql. All of the variables in the configured text are replaced with the attribute values. Precedence is from the POST attributes, after from the GET attributes. The "Encoding" is configured in DBModules in rvp.conf. Everything needed is to execute the text with the database client software functions with the correct encoding parameters. The RVP replaces empty or missing values inside ' -characters with a text NULL by removing the surrounding ' characters. If the database client software does not use standard SQL syntax, the module should convert the SQL-text to the syntax the database client software uses.
The optional auto configuration function <module name>_autoconf can be used to configure the transaction parameters otherwice set in the transactions.conf. The reading and writing operations are fundamentally different from each other. When writing, the writing occurs when the attributes are checked. The output buffer is used to queue the output before the result is known. In reading the buffer is not used and even larger files can be output without buffering. For example the module file.so can set the modifies into false automatically. Start of the SQL-statements can be read to find uncommented UPDATE or INSERT texts from the SQL-syntax to configure the transaction automatically.
To open files, the directory path in use can be copied from the attribute dirpath in structure transaction_params.
Output
The database module should return the results of the database transaction results attributes as name-value pairs separated with '=' and '&' characters. If these characters are used in the content of the values, they must be bypassed with a '\' -character. Bypass character is bypassed with another bypass character '\\'. The module should not return attributes if there is no lines in the result set.
The TAG understands comma separated values or other delimited file formats. The database module may output attribute values as delimiter separated texts, for example CSV (comma separated values) formats. Still the output key-value pair format should stay the same.
If error occurs, transaction_params structure contains the Returns -attribute name to be used in reporting the error to the output as a key-value pair. The content of the error description is printed in the page. If a system error page is needed, module functions should return a system error number from the run/etc/errors.conf. System error values are get from the return values of the function and compared to the ones in run/etc/errors.conf. If the module sends an extra attribute to the system error page, the attribute used should be ModuleError or other in the system error page.
Authentication modules
Authentication modules are essentially the same database modules with the same function definitions. If "SetsRoles" is set, the roles are searched from the module output. If these attributes are found, they are saved in the session database and the authentication is successful. Sometimes direct input is needed to use a standard protocol in developing the modules.
|
"DirectInput"
|
If "DirectInput" is used, the module receives the request as a direct input. The input is get from parameters CBFILE **param and CBFILE **queryparam. Usually these are otherwice set to point to NULL. The **param stream is the HTTP POST stream (for example the SAML SOAP would appear here) and the **queryparam is the HTTP GET string. The integrity check does not validate variables if the "DirectInput" is set. The SQL-statement *sqlst or the parameter is not filled with the declared attribute values and is in turn NULL. It is recommended to include an own configuration file in every modules use. "DirPath" may be used in finding the file locations. |
Developing type modules
The type modules are similar to the database modules. The functionality is simple. The function is called by the attribute name or by the name given in "InbuiltName". The function should return value 0 on success and other values if an error occurred.