Embedded Files
GENERAL KNOWLEDGE /
1. FRONTEND-BACKEND STRUCTURE, WORK CONCEPT
An application is web-based when partially or in-whole the application utilizes some parts of the web. Such application can have traditional desktop GUI but can use an internet browser for such function. The computing part of the application can run on PC, laptop or mobil devices but can be delegated to servers as well. Similarly, data management part can be planted on devices but also on a server. The server can be a dedicated one or in the cloud, managed by the company or the cloud provider.
As a general model, a web application stands on the next pillars:
data management
business (computing) logic
interface towards other applications or a GUI
interface towards users (GUI)
The web application contains three main parts:
data storage (usually SQL or noSQL database)
backend (data management, business logic, interface)
frontend (GUI)
In case a web application only provides service towards other applications, the frontend part is optional or non-existent.
Traditionally, the web application structure is built up from the next levels:
thin client: a frontend application usually written in JavaScript
internet browser: runs the thin application, handles communication (HTTP, HTTPS, FTP, TCP, SMTP etc.) and security protocols
thick server: a backend application written e.g. in Java on server side
2. BACKEND
The backend is built up from the next layers:
Storage (or Repository or Data Access)
Business (or Business Logic or Computing)
Communication (or Interface)
The Communication layer is made of controllers, and connects the frontend with backend. It handles requests and if necessary, does authentication as well.
The Business layer does the processing. That means, upon any request handled by the Communication level the data check already will happen in the Business layer.
The Storage layer (also called by Repository layer or Data Access layer) handles all processes connected to data storage like solo databases, cloud-based databases and files.
The layers are formed due to the need to separate tasks. All tasks connected to communication, must be contained by the Communication layer and so on. In such building when there is a change in a layer, the change will not have effect on other layers.
The Communication layer is built up from the next components:
dispatcher servlet
mapping
controller
The dispatcher servlet receives the requests from the outside, converts it to readable information and forwards them to the inner processing parts, later receives view from the inner processing parts and sends them to the outside. Usually maintained by a framework like Spring Boot.
Mapping means the information is directed by request type and endpoint to the proper direction.
The controller contains backend methods which receive the mapped incoming requests, start processing them, have the results and send those to the framework.
The data is received with a request or sent with a response.
The transferred data package is called payload.
The controller can serve payloads in two main modes:
view: the response will navigate a web page or a template along with model data
object: the response contains data without navigation, following data transfer standards (e.g. REST, SOAP, GraphQL, RPC)
Generally, at backend for different tasks different threads are launched.
The main() method starts these threads then should go to wait.
Such threads are organized into classes, these classes are called controllers.
In a controller several endpoints can be defined. These endpoints are unique, practically methods which handle endpoints coming in via requests. Overloading such methods with keeping the same endpoint is not allowed, in such case use different endpoint.
Of course, in a controller there can be fields to keep data, furthermore helper methods to increase modularity and language structure.
Best practice is to keep the controller as thin as it can be so for data checks and processing the flow should be forwarded to the Business layer where the processor and helper methods are.
It is also best practice to keep all communication processes within the controller which means e.g. any conversion processes needed to create outgoing data from generated data must be held in the controller.
In case the project is organized around different packages where a package contains its element and storage classes, it is best practice to put all correspondent classes like controllers, services into that package. In this case modularity appears in packages as well. Adding, removing or modifying package will not have effect on other packages.
In case the application would contain only one such package, it can be organized by keeping element classes in a package, the classes representing the three web app layers in a separate one.
In case the class system is complex, there can be sub-packages under the present ones for both solutions.
On frontend side, there can be three main solutions:
for visualizing data, static pages can be used
for receiving data from the backend dynamically, and optionally sending data back as well, templates can be used
for complex solutions, frontend development is necessary
A template is an JSON/HTML/XML/etc. page into which the answer is injected via defining HTML variables and frontend (e.g. Javascript) codes.
Earlier, templates could be utilized only when the controller worked in view mode only. Today, object mode can be used as well via ModelAndView (in latter case object mode is commonly called Model&View mode).
For all responses, the dispatcher servlet sends not just the response but an HTML code as well:
1xx (transient response, more is coming)
2xx (success)
3xx (redirect)
4xx (the client asked for something the server could not or would not deliver, e.g. 404 = page not found)
5xx (server-side error)
For all the possible HTML codes in verbal form, visit the SPRING BOOT TESTS page (StatusResultMatcher methods).
Building a whole web application needs much complexity in code.
For simplification, using the Spring Boot framework is advised.
3. API
The API means Application Programming Interface.
Creating an API from an application means that the application will have open points where other applications can connect and use the original application as a black box.
That means, a backend API means collection of endpoints provided by the backend to which any frontends can connect by sending requests to provided endpoints.
Following best practices, backend development of Communication layer manifested as a thin Controller class provides the API of a backend web application.
Web APIs are used for providing computing and/or backend support for several customized applications, even from other companies.
In order to use a web API, protocols must be defined.
A protocol contains that for what exact request what exact answer the other part will provide.
A protocol is created by defining an endpoint and providing request and response formulas (e.g. JSON, XML, see later) along.
For API, it is important that the interface must show that it is an API and which version of it is published.
That happens with naming the endpoints correctly.
EXAMPLE FOR API ENDPOINT NAMINGS:
For purpose, in case the API handles the tasks below, is called CRUD API:
Create
Retrieve
Update
Delete
4. API ARCHITECTURES
Data transfer between backend and other backend or backend and frontend is regulated by web protocols:
HTTP (Hyper Text Transfer Protocol)
SMTP (Simple Mail Transfer Protocol)
FTP (File Transfer Protocol)
TCP (Transmission Control Protocol)
The used API architectural style (data format) depends on:
which web protocol is used in communication
what is the purpose of the application
how much complexity the application can bear
how much time is available for development
how old is the application
API architectural styles in order of appearance:
CORBA (1991)
RDA (1993)
XML-RPC (1998)
SOAP (Simple Object Access Protocol, 1999)
REST (REpresentational State Transfer, 2000)
JSON-RPC (2005)
ODATA (2007)
GraphQL (Graph Query Language, 2015)
gRPC (RPC: Remote Procedure Call, 2016)
The SOAP makes a strict system, format is XML. Not flexible for its format but flexible in its modules, e.g. when security data should be added, the proper module should be inserted only. The conversion from SOAP to objects (parsing) can need a lot of processing from the backend and effort from developers. Some programming languages have dedicated methods for parsing.
SOAP pros:
security (supports WS-Security and allows for encryption of messages; it meets enterprise-level transaction quality and is often used in CRM software, payments gateways, and fintech solutions)
stateful operations (SOAP APIs are stateless by default, yet stateful is also an option, making it convenient for bank transfers or flight bookings where a chain of operations has to be made)
compliance (supports Atomicity, Consistency, Isolation, and Durability (ACID), an enterprise-grade transaction quality, this makes it appealing for exchanging sensitive information in enterprises)
built-in retry logic (in case of a failed communication, SOAP's response will showcase the error information so that the failure can be understood and fixed easily)
SOAP cons:
limited data format support (only supports the wordy XML structure for its requests and responses)
lengthy processing (can only transfer messages as heavyweight XML files, it will take more time to process a request and transfer a response)
inflexible (own rules and security measures which limits the freedom of the developer during programming)
high learning curve (harder to code, needs good understanding of all the restrictive rules and protocols that are involved)
REST pros:
cacheability (allows for caching of data on the HTTP level, thus, eliminating the need for constant client-server interaction, which boosts performance and scalability, this makes REST a great option for IoT-based projects)
supports multiple formats (allowing for storage and data exchange to be carried out via various formats is one of the biggest advantages of REST, that's what makes it one of the top choices when developing public APIs)
ubiquitous (the popularity and maturity of REST make it quite widespread, many developers are familiar with it and can easily work on a project that requires it)
intuitive (since REST is built upon the semantics of HTTP, it's great for public-facing interfaces that need to be intuitive for the end-user)
REST cons:
ambiguity (REST can be interpreted in different ways, there isn't a single correct way to build a REST API, practically-speaking, this can create challenges for developers)
lower security (since REST doesn't impose any security protocols, it is generally considered less safe than other web APIs; of course, the connection can be made more secure by setting up communications through HTTPS; however, that is still an extra task that developers have to think about)
stateless architecture (REST's third architectural constraint requires statelessness, meaning that each API call has to be independent of previous calls, Naturally, this might be an issue if a web service needs a chain of interconnected messages)
See REST later.
GraphQL is an open-source query language for APIs that was internally developed by Facebook and subsequently released to the public. It allows clients to request any specific data that is needed, which enables the return of predictable results.
GraphQL pros:
better performance (thanks to GraphQL's ability to specify exactly the scope of data required in every instance, there’s no excess data exchange between the server and the client, this leads to better app performance and is one of the top advantages of GraphQL over REST)
no versioning (as APIs evolve, keeping the old version while transitioning to a new one can be a hassle, GraphQL avoids this problem by deprecating APIs on a field level, allowing old ones to be removed from the schema without impacting existing queries)
detailed error descriptions (if an error occurs while a GraphQL query is being processed, the backend will deliver a comprehensive error message, naturally, this speeds up any issue resolutions)
API documentation (GraphQL auto-generates documentation on any API changes, allowing developers to spend less time keeping track of updates)
GraphQL cons:
complex caching (since each GraphQL query is unique (or almost unique), data caching becomes a major challenge; to solve this problem, developers need to implement additional mechanisms, perhaps this is one of the most serious disadvantages due to which GraphQL has not yet conquered the world)
steep learning curve (in comparison with REST, GraphQL is more niche and might require more time for developers to grasp)
crashing potential (if too many nested fields are requested simultaneously through GraphQL, a system can overload and the server may crash)
GRPC is an open-source RPC framework that was created by Google, can run in any environment, and be implemented in a variety of languages. GRPC is contract-based and uses the Protocol Buffers binary data format. It requires both the client and the server in data exchange to have access to the same schema definition. Based on the HTTP/2 protocol.
GRPC pros:
code generation (native code generation for client/server apps from Protobuf definitions to various programming languages, it helps reduce development time in applications with a variety of services)
heightened performance (binary formats of data exchange tend to be more compact than text-based ones, thus, gRPC can outperform REST and GraphQL due to its binary format requirement)
good security (gRPC requires HTTP/2 and TLS/SSL, it provides decent levels of security by default)
GRPC cons:
limited browser support (gRPC relies on HTTP/2, and that makes it impossible to call a gRPC service from a web browser directly, instead, a proxy is needed, which can come with its own limitations)
unreadable format for humans (the data is compressed to a binary format, Protobuf files become unreadable for people, hence, extra tools are needed to analyze payloads or perform debugging)
lack of maturity (finding skilled developers with an extensive portfolio can be more of a challenge, however, the slower adoption might just be a result of the technology's novelty and will wear off in the coming years)
Comparison of API architectural styles follows.
5. REST
A RESTful architecture should be aligned with the following six constraints:
uniform interface
client-server decoupling
stateless
cacheable data
layered system architecture
code-on-demand (optional)
Coming from the constraints, in case of REST API, the application will be a RESTful application, will fulfill the next conditions:
will communicate with a package which contains not just data in transfer format like JSON but header with metadata, authorization, uniform resource identifier (URI), caching, cookies etc.
the package must be managed through HTTP
the package must be stateless (does not contain client-specific information)
the package must not contain information about the path the package goes through (servers and other network elements)
The communication is built-up from requests which go from the client to the server, as an answer the server may send data back.
A request contains a URI (Uniform Resource Identifier) which points exactly to a specific target (endpoint, see later) and optionally can carry a query (see later).
Request types can be:
GET: asks data from the server in form of a query (retrieve)
POST: new data (create)
PUT: full modification of data (update)
PATCH: partial modification of data (update)
DELETE: remove data (delete)
OPTIONS: asks for meta-data
HEAD
In the URL, the carried data of GET requests (queries) are visible whilst the carried data of other requests are hidden and can be sent via the frontend or cURL command (see later).
None of the request types creates obligation: the backend can render any code to any types; using the right type for the right task increases readability.
The incoming requests are handled through methods developed by developers at backend side therefore it is allowed to use any requests for any tasks. However, there are differences and best practices:
the GET query is always public data so sending sensitive data via GET query is not good, asking data is preferred
the other requests are secure, the data is coded, sending data is preferred
some schools say every request could be sent via POST request however, differentiating between requests can be done by following the request type
It is also possible to send request body via GET request as well however by using this option the purpose of GET can be injured.
The communication is always established via using URLs. The different network elements, DNS servers handle the targeting.
If the server is configured properly, before a request reaches the web application, must go through different levels of security (e.g. firewall) and web services.
Before POST and PUT requests, one of the browser security protocols sends an OPTIONS request (pre-flight request). In case the web application receives the request, it sends meta-data back like in which format it communicates (e.g. JSON, XML, plain-text, text/html).
Usually, the same company develops the frontend and the backend applications therefore the frontend is prepared for receiving one format only which was chosen at the very beginning of development.
A request has the next formula:
base path (protocol://server_address)
path (/endpoint[?parameter=value[¶meter2=value]...])
The endpoint defines a group under which the different request types are checked at backend.
In case there is no endpoint or under it no request type is defined in the web application, the backend will not send answer to the frontend.
Best practice for endpoints is that queries should be used in case of possible more dependent elements in the answer.
EXAMPLE FOR A REQUEST:
The typical manifestation of a REST CRUD API can be seen below.
6. JSON COMMUNICATION FILE
The possible (accepted) requests formulas must be defined in order to limit the communication possibilities and introduced standards between the frontend and the backend.
In case the controller methods are working in object mode, such rules are usually defined in JSON communication file.
JSON: Javascript Object Notation, data transfer standard between internet elements such as servers and clients.
This file should be named as JSON.txt and placed into the src/main/java directory.
Elements of JSON format:
outer {} marks an object, if there is not, marks a literal (no variable or array it can contain)
"variableName" marks a variable name within the object
lesser-greater signs: mark type and format information for developers only
double quote: variable name or text value
|: separates possible options (typically for enums)
[...]: array (because a JSON file contains data which has fix size, no List is available in JSON)
[{}, ...]: an array where an element is described inside the inner {}
the format is not sensitive for spaces and line breaks; the automatically generated JSON contains neither
Values in JSON:
numbers: 12.3, -36, "25.4"
enums as text: "CARD", "CHICKEN"
logical: true, false
text: "abc", "ooo"
null: null
EXAMPLE FOR JSON COMMUNICATION FILE (FOR DEVELOPERS):
This file can not be automatically used since contains human-friendly data as well but can be modified easily to be proper. In this form informs the frontend developer about exact data formulas.
EXAMPLE FOR JSON COMMUNICATION FILE (BETWEEN SOFTWARES):
The JSON is visualized better if formatted with spaces and line breaks. Such format is shown by browsers for better readability.
EXAMPLE FOR A FORMATTED JSON COMMUNICATION FILE (LITERALS IN ARRAY, SPACES):
EXAMPLE FOR A FORMATTED JSON COMMUNICATION FILE (OBJECTS IN ARRAY, SPACES):
EXAMPLE FOR A FORMATTED JSON COMMUNICATION FILE (LITERALS IN ARRAY, SPACES AND LINE BREAKS):
EXAMPLE FOR A FORMATTED JSON COMMUNICATION FILE (OBJECT IN ARRAY, SPACES AND LINE BREAKS):
The serialization and deserialization processes which convert data to and from JSON are language-dependent but the JSON itself is platform-independent, must work under the same rules whatever servers and frontends communicate with each other.
Before processing incoming JSON data, it can be necessary to parse it to a useful format like Map. See the JSON PARSE page for examples.
7. API TYPES
Open APIs (also called Public APIs):
accessible to anyone on the internet
typically used to allow integration with third-party applications
examples include payment gateways or social media APIs
Internal APIs:
designed for use within an organization
allows systems, teams, or applications within the organization to communicate effectively without external access
Partner APIs:
shared with specific business partners
often used in collaborations or partnerships to exchange data securely
Web APIs:
used for communication over the web, typically following HTTP protocols
includes RESTful APIs and SOAP APIs
typical in gateway-microservice communication
GraphQL APIs:
a query language-based API that allows clients to request specific data structures
used for improved efficiency, especially in modern web apps
Composite APIs:
combine multiple APIs or services into a single call
useful for handling complex processes requiring data from various sources
Hardware APIs:
enable communication between software and hardware components
common in IoT devices or system-level applications
Library APIs:
provided within programming libraries/frameworks
often used to simplify tasks, like data manipulation, graphics rendering, or machine learning
Streaming APIs:
deliver data streams in real-time, often used for continuous data feeds like social media updates or stock prices
In a usual RESTful microservices architecture
the API gateway communicates with web API over HTTP/HTTPS (Angular easily access it), handles routing, load balancing, authentication, and orchestration
the microservices commonly communicate with web API on RESTful design (simplicity and widespread adoption), or gRPC API on performance requirements (efficient binary communication)
8. CURL
The cURL is a PowerShell command line app for sending requests.
Ideal tool for integration test of an application.
A cURL distribution can already be found in Windows and Linux and the curl alias is pointing on that. If that distribution is not acceptable (it is not updated), install a fresh version from the internet.
In order to let the external cURL app being available, first the alias must be removed before the new cURL can be reached:
remove-item alias:curl
No message will appear in console in case the alias existed and removed successfully.
The GitHub webpage presents a public endpoint for developers to test whether the cURL is operable:
api.github.com/octocat
Important parameters:
-V or --version (get cURL version)
-h or --help (get help)
-d "data" or --data "data" (for POST, PUT, DELETE, ... request body)
-X <request_type> (overwrite GET/POST request type with GET, POST, PUT etc.)
--write-out (extract HTTP response code)
-H "header" (use given header, means incoming HTML attributes)
-i (show request body along with response protocol headers e.g. HTTP version with response code (200, 404 etc.), content type, encoding, zoned date and time)
-o <file> or --output <file> (output into file instead of console)
-T <file> or --upload-file <file> (send local file to server)
-u <user:password> or --user <user:password> (server login data)
-v or --verbose (talkative output)
@<jsonfile> (use JSON file, no outer quote signs in it, in case there should be, use --upload)
The default request type is GET (no need to define it explicitly).
In case -d (--data) parameter is given, the default request type will be POST.
In all cases the needed request type is not equal with the actual default, must the needed request type be expressed.
Order of parameters can be any.
For switches starting with - sign, they can be written as one parameter, order is unimportant:
-Viv
Header can be:
"Accept: content-type"
"Content-Type: content-type"
The Accept parameter defines in which format the cURL waits response from the server. In case the server (willingly or not) is not prepared for that format, cURL will receive an error HTTP message.
The Content-Type parameter defines in which format the JSON will be sent to the server.
For formatted JSON output, use the cURL command with jq pipelines:
curl ... | jq (Windows)
curl ... | json_pp (Mac, Linux)
Sending JSON object in command line with cURL should follow specific formulas:
"{"variable": "value"}" (general format)
'{"variable": "value"}' (mostly on Windows)
"{\"variable\": \"value\"}" (escaped format)
When the JSON formula is read from an external file by cURL, the general format is acceptable.
In order to keep readability for the command in command line, when typing, use the ^ (Windows) or \ (Linux) character when a line break is needed.
Testing endpoints is possible by e.g. write batched cURL commands with JSON formulas in outer files and checking results against expected.
The JSON in external file can be either un-formatted or formatted.
EXAMPLE FOR SENDING REQUESTS:
9. CONTAINERS
A container is a unit of software in which application code is packaged alongside all the files necessary to make it run. This makes it easy to move the contained application between environments while retaining full functionality.
By utilizing container techniques (Docker, Kubernetes) it is possible to change application parts (microservices) along keeping the rest of the application intact. Therefore, it is perfect for management, scaling and deployment.
Also, using such technique means that on server or cloud side, the application is operated from these containers, instead of managing all members one-by-one.
Google Sites
Report abuse