The Albion Sofrware Journey (Work in Progress)

The Albion Journey:

It started out with a Laptop and extra screen with Visual Studio and code checkout from SVN, later migrated to Mercurial for concurrent working and in a short space of time git, because it was just so much faster at the job. The Build system shortly after the migration to Mercurial was migrated to Tundra and, that could calculate the DAG and build concurrently with threads, push 24 thread to max out all IO and CPU on a machine for 4 minute scratch build times down from very long 30 minutes or 45, absolute significant improvement. We self purchases SSD 128GB  ourselves, while ever one still they fail extra, because the time saving costs was  knocked  it down even further due to all the random io. Given everything in source and re-install a whole machine from scatch in less 1 hours on an SSD. When done this how many times know 25 digit product keys off memory, there no point worrying about drive crashes as you can do it in your sleep, easpecially when everything is always incrementally back up ones work. At times I used to have 4 desktops open, each correlated with the different combination of windows on a seperate problem and branch checkout for albion. The reason re-compiling was slow after checkout swap, running 4 concurrent folder streams keeping the same cache was must faster when flipping between things and saved more time. Typically one was correlated to admin. Task bars used run verticle, so differentiate windows problem of similar name, that all look the same in thumbnails and alt tab is not what used  to be like in windows 98, xp...

After launching the program for the first time and opening the first test dataset, I was crashing the program with every click. So I got to work and started debugging the code in the grid, so I could hopefully perform some click actions without everything crashing.

Grid and Attribute Editor converted Relative Pixel Mouse to grid to Row, Column and would then read the data directly from the in Memory table using the index numbers. Sadly, there was not boundary checking for the last row and last column, so every time click window/remaining unused space the grid/attrib/project explorer crashed. So I then implemented boundary checks for the inMemory Grid and the SQL driver based grid to cover all instances. Starting to get somewhere. Additoinal was all the different sync modes, so you could open mutiple grids at once  and  tweak the selection syncronization between cad /GIS to how need to operate to solve you r issues.

When I start the grid was rather simple and plain, not much functionality, just a grid and table selector.
So I started out knowing everything was opening large datasets, I immediately got to work

and started background colours for cells, and layout views with inheritance, using bit flags.
This allows one to set up and save versions of field display configuration and layout, which were visible and not, in which I wrote the configuration out to the local Application Data folder of the users PC. Actually turn out to be a little more tricker than initial thought, because the grid had slight not nice code layout, so like 2 entry points in which wire this in for the field re-ordering and visibility through the database driver, but was complete by chatting with my colleague to better understand that sections caveats, which he  was struggling to remember. As the grid had two operating modes in memory only and driver/adaptor mode.

Things were not very faster at the time and PC were slow, so I then optimized a few of the stats calculations that were displayed for a selection, to reduce the computation overheads and be more optimal, selections large than a certain limit I imposed limits, that you have to click to compute the results, especially with large datasets.

Improvements were then made to the SQL bar for interactions to be driving by the keyboard and not just the button.

At a point in time personal didn’t understand why certain rows didn’t join or render correctly, this due to directly ASCII, no printable characters, so we implemented concept of bad cells, which colour the background of the cell in light red, so user knew which cells had trailing or leading bad characters.

We then implemented additional tooling to assist users with clean up of the Bad Cells and, so they could fix them if necessary.

Follow by all the Fixing that you see discussed in the Grid and manuals.

At certain point reporting came up on how can we do reports, because currently they supplied raw data to another provider that then did the markup and made it look pretty.
I then said, well lets expose SQLLight which was already use to wrap the internal memory write cache modtables, which partially used already to filter things on the grid fully, so we could write SQL and generates reports for and with in Albion by transform the resulting dataset into HTML 5 and we embed a browser.

That is exactly what we did my colleague got chrome bedded in Delphi, I work on exposed and improved all the SQL integration with Albion, so they could write SQL reports and execute them against the InMemory Tables and Drivers and we then exposed a few different functions that generated the table names using different conventions to assit them and make certain things more abstract and simpler for some things, based on the current open connection adapters. My Colleague then got to work on re-writting the reports over to SQLite and written generic transform template from InMemory Results dataset to HTML5/CSS. We had some fun challanges to  gether to figure out how to redo some of the reports, as not  just perfect as we had do some tricks to  get it into the table final datashape that was need.

I continue to enhance the reporting with progress bars and cancel buttons and implement a unit test to ensure and start to improve the attribute cohesion/conversion worked correct expected with the matrix of conversions unit tests.

A few months more down the line things started to get a little crazy as I did not have the same software version as myself and everyone was now calling me to assist with issues or asking how I could do this or do you know a better way to do that. I was spending alot of my time just supporting the internal staff with things. I did do exercise at once record the amount of time I think 2 hours a day assisting staff, because I was not getting enough fixing done in my personal opinion but when presented to business or management, however, no had the slightest interest. Not oh that not good or we might need training extra, what type of issues, nothing came from sharing that information, just dull looks.

I then implemented and improvement to the field configuration tool, because re-organizing columns, became tricky at times, and was pain full, so I implemented Ctl to select multiple items and what I called collapsible row drag, for the Grid layout Configuration thats cells were directly create and place in memory and not abstractly load via a driver. This actually proved to be rather tricky having to snapshot everything before and then allow things to incremental move and drag under new groups as all raw positioning. At the end of the day with some testing all the issues were found and glitching and it was released.

Many further improvements were made as you can see in the grid manual.

I was then approached Danie to implement the Verinoid Algorithm to assist him and a data comparison tool, that tell which records had been created and deleted and join the information up to the resulting dataset, so could easily assist them. I then build that tooling into the grid.

IT then wanted the space bar shortcut key implemented to make column selection easier.
I then went on spree of implemented all know shortcut keys combinations and choosing to align with Excel as these all internally know, then improve on what excel had to offer with better logic than them and ability to Search implemented by finding next value of cell equal to my current one, and then modifiers or or usefully variants I could then of. Such when shift was or was not hold, stoping on the end of the last item or stopping on the next not matching item or and things like that. While doing this keeping in mind the 4 directions up/down, left, right that would complete all set and permutations of keys that would bee needed. At one should be able to operate at speed and easy on the keyboard now with less click and trying to manual make selections of corrilated information.

Additionally I then add improved selection copy and pasting, that could detect adding more rows to the end of a table and prevent that happening if a table was read only,
user were provide more information based on the type of situation to assit them in understanding the constraints limitations of selection being of miss matching sizes for the grid..

Later it them came down to adding Sort routines, which required me to go modify and enhance the database attribute comparators and Implemented optimized code for doing case sensitive, non cases sensitive and numerical conversion and prefix numerical sorting and think some special null order handling as well..
In the background, this meant Numerical sort, had to convert text to numbers and then sort it properly, so I wrote number detection and extraction from the text such it could be numerically prefix sorted. Additionally I was consider and had consider look at things like Radix sort to speed things up at the time, which I had previously been through  based on algorithm I was looking into on the internet  while studying that were not covered in the Curriculum at all on weekend, typically many other by self choice and interest after high school.

As things continued, more discrepancies and many different forms of bugs and problems start to present themselves from staff, I continue to improve this to reduce and remove any form of confusion. I then added different grid colour highlighting, based on with CAD item had labels rendered or not or geometry rendered or not or had null geometry or no geometry.
As the staff started getting really confused, with all the dataset that they had open and some records they not able to find in the CAD at times, thought was bug in the software and we start to find that users would share saved project files, so someone turn on rendering styles of certain things and others would turn it off. Start giving me alot of false positives of personal there bug were is my stuff gone. So I improved the grid to communicate to the operate clearly the state of things, If there were labels that were render for the item if there was null geometry or geometry or shape was actually rendered. This way there was no less confusion as to why nothing was graphically render or if an item rendering schema, didn’t match that of selected record. It just easier to debug things less questions.
Improvement of grid status communication and how narrow refiltering filtered results of the grid work.

I later then got into the Rendering engine pipeline and start to improve things there by adding additional functionality or missing stuff and then again for Pipe Inspection module. Then many small issues and tweaks and improvements to speed up setup of the rendering settings.
Then additional label placements. I do remember the code had many different permutations and combinations for rendering, faster shortcuts for setting up directly from the explorer the ability to change colours and skip many clicks.


Making the Operator more productive and less time waiting time tricks, while the operator blinks their eye when focusing in on the next step and time shifting the back heavy computation into the eye blink, which allowed the next step brain thought process to be triggered.

This was done by ensure interactive grid would immediate render sub milliseconds and then on second, 3rd rendering loops, other heave stuff.
This means as the selection finished, the user moved to the next operation thought or blinked and look at the CAD for the result, by the time they blink/refocus on the CAD was upto date with the selection. This prevent the grid feeling sluggish and slow, as have immediate sub 1ms response for rendering, speeding up the user next step thought process.

Further Speed improvements to the grid were then made as well, when scrolling it would display items that were already cached in memory immediately, however, items that would have to hit disk or network to display were skipped, this was to speed up the users experience and make things faster, upon the user stopping with 20ms pause, it would trigger the fetching of data in the background which then be displayed. Further more the scroll bar, just as I later did at DotActive later again as annoyed me, was a tweak to be fast at drag scroll and allow the pull of the bar down super fast only display what was in memory and then upon slight pause fetch disk and network, this allowed to locate records with speed, otherwise, get stuck trying to slowly and painfully move down the grid, your best bet was to basically search for an item.
It was a terrible experience, slow and unproductive at best.

Furthermore I then separate the grid selection that would hang while the CAD and other status were computed for the selection and hang for like 0-0.3seconds. I allow the grid selection to render first and then post a delayed selection event via the windows event Queue, allow for repaint of the grid and then for the window event queue to be processed again.
On the delay selection event I subscribe all existing sync events to the CAD system and status computations. I then went even further and implemented further delays like throughout the application, I placed all the trigger events that other applications written in Delphi where listening for also onto the same delay mechanisms. Additional Albion CRUD event hooks were given preference and then on delay dependent software, this was to ensure that no one could hook code into the event loop that would hang or slow down the speed of the Albion software.
Bad code written by plugin applications would remain in plugin applications and their app would then be slow.

This worked really well, because when the user was focus on the grid to make their selection, it would reflect immediately, they would then change their eye focus to the other items in the CAD and status windows and by the time ones eye blinked if they really fast, they just maybe notify the screen to refresh. Generally however, they won’t pick that up and would be updated.

This work well, because every user grid selection made the user work faster, because they no longer waited 0-0.3s, grid be immediate and they get to work and be more productive on there next steps.

Just imagine the adding up all this productive time saving in grid scrolling and immediate selections repaint, next step brain trickery.

6 months in and well, with all the bugs that still present and like there new features extra, it was time to full automated and full pipeline and build from start to every single machine in the company. Using NSIS, I re-did all the installer packages, track down and sorted out the mess of dll's were some not used, some belong different packages there internal company costing files being package into installers, sperated everything into there respective layers and then automated all the different variants of the products.
They would build at 12pm at night, and by 6am the next morning all the test would have been completed, the preparing of all files were bundle into correct folders for all the software variants, then organised with IT to have a login script in the moring overwrite the previous version of software every time.

On top of these I ran mutiple combinations of the packets. Stable and Latest and then combinations of stable and latest of the depent dlls, so 4 copies of program,
which scripts that any one could run, to check out any combination of albion and the delphi code. Additional archive copies of everything were kept for ever since day,
so could easily revert to any version that was build upon any day. This increase the productive, because on click button I could release a new version to the company, just send them a link to run a script and they would all have a fix with in minutes. Most importantly we not packaging internal costs files into install packages meant for the public.

Then eventually came the time, were it was december and I was totally  un happy with the  crude method of  communicate records across to delphi.
Were if  was to big for the pre alloacted  array of these, just drop the results of the records. How  could one function with this like random prob and just hope there less records that what you have requested. So said to my colleague that working over christmas period with myself, I want to improve this, lets work to gather to get  nice mapping of the information to delphi from c++. so basically wrote additional functions to transfer all required information to delphi and the  explain and help colleague write  new wrapper class to represent  the cc information in  similar  mirror class to what we had  in  c++. This then was  how we moved forward in a much more efficient , accurate.

Zero inhrited flaw crash, of many, In Memory Storage of Optimized Custom Heap Attributes(MSOCHA) for Column and String Caching in ModTables, had compression applied as quick way reduce memory, Upon saving it would crash as it, as would uncompressing all the data to save it, defeating the purpose of compress it in the first place needing even more memory than when one compress it in the first place.. Wrote a Iterator into the whole MSOCHA iterate the pages and bits flags, so update save to iterate through MSOCHA with small buffer overhead and incremental save changes and write/flush the changes to the underlying Database. Avoiding all the memory issues and  logicial flaws in thinking.

Then I started to get into the GIS Network Stuff and fixing issues around triggers and bugs in that section that allowed node and polyline dataset where the nodes were stored in one dataset and the polylines in another dataset, to act as one. Alot more improvements were made to this where I wrote improved UI tooling to work with my Pipe Inspection Project, to make capturing a pipe as easy as could be with modifier shotcut keys to assit users in certain operation. That was really fun, rember the first simple go  just between 2 nodes and polyline work, out super  quick easy just like I plan, but then when realizing I need to  run on much large scale, the whole arhiteacture had to change to address or the 5/7 corner cases.  Additionally  the amount of data and the way it had to be prepared and assemble in memory to allow tool to work had to increase, trying to not increase memory, but eventually was the only way I could capture all corner cases with better upfront loading and pairing of information.

Then were issues of rendering failing, were errors were swallowed and never communicated out of the render, after much debugging was determined that the projections were failing. I then added additional column to the layer editor and add the projection icon, with different colours, to highlight if there were errors. Upon testing all work and went well and this would saving me from that problem again. It was deploy and then well was one of those days when some things went wrong, I went to attend to someone saything things were really slow and well, then narrow it down to the change I had previously made. Advise was then set out to revert to previous days versions.
It became apparent that the class and API were very poorly layered and this what lead me to have further ideas on how code should be layer and optimized across different L1, L2, L3, all impedance mismatched. So what the issues was, is upon reading a simple class variable of valid projection, it was causing the disk to be hit and reload the project file everytime the UI wanted to render. this was one of the side effects of having and SSD and others not having an spindle, it didn't notice the lag at all when testing.
I then improved the layering of the program, to correclty cache the resulting of projections such that things didn't load from disk in memory, such that rendering painting would basically only be accessing memory.

This little issues lead me to work on class and laying constraint technique, that I wanted to avoid this, so idea was that everything you call should have a performance requirement and if violated the code shouldn't compile and one should ideally never be able to jump more than a single performance impedance mismatch.
Thinking of L1, L2, L3, Ram, disk and many other network IO impedance mismatches. When writing a class, you write a class and specify its performance requirement spec that it had to meet for those consuming it and additional the requirement from which it would read all its information from. This could ensure and would ensure that this type of problem with bad class layer design would never happen and could be assured that everything abided by the performance constraints, without having to go inspect and read all the code through every layer.

GIT concept source control or Change Control System for GIS

The product had a simple CRUD operation tracker written into the software, to operate basically just like source control on a flat database single table at a time no relationships. But soon users end up directly open these database with out using the change control system and well caused about 20 000 records to be corrupted. The company and data captures had to basically go find there last dataset that they worked on after a 2 week attempt to recover or the identify the records that broken between the users, but nothing of the multiple attempts and approaches I took could I recover there dataset, to them it was still pretty wrong. So they had to start from there last good backup. Additionally  I was not involved then there issues, that had done restores of the postgres database  a week or two earlier.

So since this was another lovely zero day flaw bug in this software, I then implemented fixies, to ensure that no one could circumvent or open anything that worked through a change control system directly with the software and wire in checks all over the software and did testing to ensure it would be impossible to happen again.
This followed with may other updates to the database system  migrating  from the old version of postgres to the lastest version at that point in time 9, which was a challange  as well trying  all variations of switch, to find the combinations that would export and import with out any random errors at the end of the day.

GIT source control concepts for Database and Change Control Approval Processes
This is rather simple if I compare it to what I now at 'Pattern Matched Technologies` strived to write not in a Desktop application, but in a cloud application running on GraphQL NodeJS and AWS Amplify. Were I wanted for our Product Catalogue to have the concept of git source control, so we can track who is changing pricing and everything like that and have snapshots of who did want and go back in history because we dealing with finance and money. Has many up and downstream intergrations rest APIS. Nightly scan of the upstream services and automated agents checking the integration dataset information being returned, decided if it is to auto merge or manual intervention. So here I designed/wrote the database and edit, diff merge mechanisms and design how UI for browser would need to be written to hook into all of this with adaptor type approach, few design iteration determine optimal location and were and how I should have it all wired, pros and cons, complexity and what go wrong. try make the UI implementation as simple as possible. It can track changes made to relational database with multiple tables and hierarchy 10 at least at this point in time, some are 1-1 relationships, some of 1-many and other many to many between the tables. Additional there are tables that don't run under change control that have relationships with change control tables as well and visa versa. So we have, special editing tool that populates the workingset with the changes the users is making and differing tool between any two instance of the database or subset of changeset records. Then there is a review process and an approval process, followed by merge. All the tools have extensive unit tests with full dataset written on them, that run live against the database, because there loads of triggers in the database to ensure no mistake from the APIs CRUD and additional no can introduce data schema breaking changes. which happens at this level of complexity and any direct external SQL manipulation later won't break the relationships. The tool has been extensively tested and code harden with large dataset I had already and confirm no bugs or issues timeouts extra. The only way I can improve it is moving to C++ where I can better optimize my iteration algorithms as JS is limited for this non recursive and stack based for performance, is to make them run partial sections of the dataset diff/merge changing concurrently to speed things up and attempt to do batch in memory processing reducing db latancy, if possible 50/50 here make things extremely more complex as it was consider when being written already, but then the time requirement and amount additional complex abstract thinking when well need to have been written in less than a month, so many other bugs different products to fix, hardly get to do new products work.
The Minimal viable product is now running in production and getting some few need extra improvements now on filters and views of information to make it a full fledge product.

When working with the GIS software between nodes and polylines there are some nice, ticks that emerge when dealing with this type of situation when dealing detecting and resolving merge conflicts to make this more granular than the basic, CRUD ops that Albion only handle. No intelligent vector inspections, I leave it there room for improvement.

Non recursive and stack based for performance (Linux-BRFS)

Previously during the covid period I did some courses through Stanford as I wanted to get into more AI as I like see all the possibilities then and then later took to optimizing permutation algorithms(brute force)(tail recusive basic repeating pattern in symbols, didn't find it) and then additional quick sort and depth/bread first binary tree search. From doing this in JavaScript then going to c so I could further optimize and the Linux, I identified a few more optimization for the c compile that I suggested and submitted to the C++ standards forum and next iteration of C++ forum. The performance of these algorithms, compare with stock standard implementation were amazing adn difference between 32bit and x64 architecture performance different on totally different scales, 32bit nearly double the speed than more modest improvements of 64bit. I was well were can these better be used and started to take a look at BRFS disk storage code for Linux, but didn't seem like I could find a section code that could benefit with a refactor that use the algorithm in such iterative form, upon which I could put a pull request in on Linux BRFS for some nice performance improvements on different architectures. Guess since I am in South Africa the only thing really left was to share my findings of which some AWS interview employees knew of at the time the results as looking for work. The only I guess person I knew of was that potentially would appreciate it Linus Torvalds and Linux community and would know of some other place it could be used. So I look up some contactable information and reached out by sending a few emails hint at the direction into which to write code and the performance increase that can be experience extra.. 

Then with telementary and things I could still see there were crashes and could partially understanding the state corruption happening causing the crashes.
Because this one larges GIS Verb sync system and  that was then integrate into a separate CAD system. I  had to study the whole system and make  careful notes of all the interactions at different points and then understand and identified the mistake that cause state to corrupted across the system. I then looked to which statements I would re-align  and how to fix the system. Is this way or that way better, pros and cons in the long  term maintainability.  Long might be easer and simpler to maintain, short term, might be quick but going to be  hell later, choose your poison. So after writing down and double checking everything twice, because making a mistake here  going to  be catastrophic, I went on to implement the change. 

Later on this sync verb system, I mutated into basically full database transaction system for in memory system support all index violation checks extra.
The Join Tables I then upgraded and added more unit testing for all the pointers and things, that translate sync verb system from underlying table up to top level single join table and down again. This would include index violation checks extra as Wadiso and Sewsan and Swift were making use of Join Tables to load all the data into a single table, due to database table column limitations they partitioned there data. These operations all had to be really fast, yes there were bugs to fix and improve in this space.
Additionaly alot more work I think had be put in here for the Network Topology Node, Polyline tool to work properly with the Join Tables properly.

Wadiso 6 - Next Gen

build on Delphi via dynamic C dll API Function calling layer into C/C++ Albion/Otuka core. With Mapping generate from c to Delphi function signature generation that wrapped C dll. Providing core CAD and GIS,Database,Table Grid, Driver, Undo functionality.

Bugfixs:

Drop all work aka PIM

- Column Data was Partitioned into Multiple Tables and that post join on load joined using Albion In Memory Narrow Join Tables, had a bug, records were noting being joined up correctly and were wrong.

identified issues to be Wadiso. Refactor and post fix to all logic and database function to correctly read records via the Albion In Memory Tables and not directly access children tables with the assumption of record order being the same. Added Additional logic to model creations/imports, to purposefully add dumy offsetting  records Table Partition, 3, 5, 7, 11, 13. So easily identify miss-alignment and know which partition it was, incase this was code, that was not found, circumvent vent all fixes, so fix those too in future.

- This probably stemmed Zero Day Inherit Albion Join Table bug, previously fixed year ago, where I solo, in short the (left/right) pairing was (left,left), (SRC,SRC) not (SRC, DST), (left,right), that identified from DB trigger interaction the previous colleague on Wadiso working on and lead to my week of long late 2am nights at office, urgently tracking this bug, used functionality, inspecting and debugging and validate all existing Custom Paged Memory Structures and code for bugs, to confirm 2 twice, with multiple fail attempts, reset, step past, extra, can't inspect, tweak inspection windows, that there were no memory pointer, bit page manipulation, algorithm, storage bugs.

2nd Christmas, We Improve RecordSet Exchange between Delphi+Albion C, class wrapping matching dynamic vectors of C/C++, not just truncate recs

Assisted previous colleague:

- SQL Water reports

- Generate SQL to pull Data, to post splay it into Pivot format for Excel UI reporting tool

Swift 6 -Next Gen

build on Delphi via dynamic C dll API Function calling layer into C/C++ Albion/Otuka core. With Mapping generate from c to Delphi function signature generation that wrapped C dll. Providing core CAD and GIS,Database,Table Grid, Driver, Undo functionality.

Assist with development and new functionality GIS, SQL, Grid, DB Triggers functionality to be developed in Albion/Otuka Core.

Bugfix:

Crashes:

1. Geometry GIS Issues, introduce from IMQS Otuka, that caused application to crashes.

2. Never Ending RunTime of Procedure, Memory Crashes, planned tweak to data storage caching warehouse data read for SQL column cache range prediction fetcher, to speed up data retrieval and reduce read overheads, Implemented in IMQS Building.

Dropped all current Wadiso, Pure, work.

Tested previous app for issues, check memory and states, then Change History see recent changes.

Catch Predictor had bugs, cause it to identify columns being access for current routine, so could prefetch ranges records and a set of columns more efficiently. Sadly, it could never reset the pattern, detect reduction in the pattern and would get stuck causes excessive over reading of columns. Get into the caching SQL code identify issues and tweaked existing and added in additional logic, to detect a pattern change and reducing pattern, to reduce column fetches. Previously had to be fixes to Caching systems, as there was no cache LRU or LFU eviction policy, so cache consume all RAM eventually.

This improvement reduced 80 ode Gigs of disk I/O reads to 12Gigs in post testing and monitoring to the prior Swift App, lead to significant improve in performance of Swift routines running over large dataset in millions of records and all GIS applications, Wadiso, Sewsan, Otuka.

build on Delphi via dynamic C dll API Function calling layer into C/C++ Albion/Otuka core. With Mapping generate from c to Delphi function signature generation that wrapped C dll. Providing core CAD and GIS,Database,Table Grid, Driver, Undo functionality.

Assist with development and new functionality GIS, SQL, Grid, DB Triggers functionality to be developed in Albion/Otuka Core.

Bugfix:

Crashes:

1. Geometry GIS Issues, introduce from IMQS Otuka, that caused application to crashes.

2. Never Ending RunTime of Procedure, Memory Crashes, planned tweak to data storage caching warehouse data read for SQL column cache range prediction fetcher, to speed up data retrieval and reduce read overheads, Implemented in IMQS Building.

Dropped all current Wadiso, Pure, work.

Tested previous app for issues, check memory and states, then Change History see recent changes.

Catch Predictor had bugs, cause it to identify columns being access for current routine, so could prefetch ranges records and a set of columns more efficiently. Sadly, it could never reset the pattern, detect reduction in the pattern and would get stuck causes excessive over reading of columns. Get into the caching SQL code identify issues and tweaked existing and added in additional logic, to detect a pattern change and reducing pattern, to reduce column fetches. Previously had to be fixes to Caching systems, as there was no cache LRU or LFU eviction policy, so cache consume all RAM eventually.

This improvement reduced 80 ode Gigs of disk I/O reads to 12Gigs in post testing and monitoring to the prior Swift App, lead to significant improve in performance of Swift routines running over large dataset in millions of records and all GIS applications, Wadiso, Sewsan, Otuka.

GIS Pipe Inspection Gas/Water - Pure Technologies USA - Xylem USA - Functionality

Build as App Variant on Albion in C++.

Solo Project and Lead, was Software Architect, Database Architecture and Design and UI designs and CAD Tooling features designs, user interaction design and implementation, Design of CAD themes and implementation. Developer, Programmed all supporting routines for functionary and alignment features. Wrote Documentation and Manual, had a new colleague checking documents for spelling, then bundled it into software and build installers using my existing revamped abstract installer builder for all applications variants and sent the final installer package to Pure in the USA.

- Generic Import Tools, Dialog Capture Robotic Dataset Configuration and mapping to abstracted format.

- GIS CAD alignment tooling, for constraining different pipe sections to know Lat,Long and existing pipe features. 3 hours versus 1 day estimate, followed by myself testing it in the USA and the reset of the week having late nights to 2am in hotel + break to watch independence day fireworks rush at 5pm + Sat, for all 5/7 corner cases, which required identifying crash or bugs.To solve each required a re-write of the CAD tool relook at preparation of all data lookups and required projection loaded into a structure way to allow the CAD to be able have all data need to different permutation the tool be used in, i.e node align to another node within 3,2,1 segment of each other, start and end Majors, as well when skipping nodes and Major nodes start and ends of datasets.

- Grid display import errors, running in the background had a full Error Log roll up process, that could provide context and a list of imported record errors, to identify the issues, in case some later added field constraint or trigger, that could cause a record to not be imported(error), a grid would display with providing all the context to know which record and what the issues was, so could be fixed and re-imported.

GIS Pipe Inspection Gas/Water - Pure Technologies USA - Xylem USA

Build as App Variant on Albion in C++.

Solo Project and Lead, was Software Architect, Database Architecture and Design and UI designs and CAD Tooling features designs, user interaction design and implementation, Design of CAD themes and implementation. Developer, Programmed all supporting routines for functionary and alignment features. Wrote Documentation and Manual, had a new colleague checking documents for spelling, then bundled it into software and build installers using my existing revamped abstract installer builder for all applications variants and sent the final installer package to Pure in the USA.

Features:

- Interpreted all their various supplied Robotic Datasets and design an abstracted single format Database persistent storage schema for everything

- Generic Import Tools, Dialog Capture Robotic Dataset Configuration and mapping to abstracted format.

1. Existing Water/Gas Pipe line.

2. Robotic Inspection datasets, that would be overlayed and aligned to Best known GIS/Lat, Lon pipe route.

- GIS Visual generation algorithm, that would generate GIS Geometry for the Raw dataset.

  Having different modes of alignment for compress/stretch, non adjustments left aligned.

Computed all the relative errors percentages and adjusted relative size length extra.

See more details:

https://sites.google.com/site/wiprogamming/Home/pipe-inspection-gaswater-pure-technologies-usa-xylem

GIS Browser Maps - Ablion/Otuka Title Render (Think Google Maps)

Web need Wadiso Water/Sewan reports in the Web Browse Editor, had all be re-written to postgress, but need them like yesterday for Client, so we made command line version of Albion, supplied it with a connector and the report, to for the re-exposure many reports in Wadiso/Sewsan reporting build on Albion->SQLLite->ModTables->SQL Driver->Postgress-> Generated Reports using HTML 5/css with imbedded browser, now served by an HTTP API

Albion/Otuka GIS

Core package to Suit of Products

My ideas and implemented.

- C/C++ Telemetry, Crash Dumps - Implemented Browser HTML 5 Notifications for new crashes,

which triggers, prompt investigation of the crash, followed by run up the stairs to operator, chat with them,

to better understanding UI interactions, that initiated the crash, capture all dataset to reproduce it.

I would make a call, If blocking fix immediate, else backlog list.

- Grid: 3,5*,7 Row Gradient(I struggle to follow rows), Layout Views(Inherit), Collapsible Drags, Bad Cells, Sync Control, ShortCuts, Multiline Text

- Too Many bugs, and staff not on the same page as me, repeat bug reports. Rework our deployment process and Rework our NSIS install package, to support Internal/External Variant distributions. Daily Builds and Archives of Albion and integration with Sewsan, Wadiso to be produce onto network drive, with IT setup logon script to copy the latest version to of each software to there PC. With fast reversion scripts for Combinations of Stable Albion, Last Sewsan/Wadiso, visa versa, just kickoff deploy script or install build.

- Unit Testing all Attribute/Value conversion/Cohesion data types for ModTables an SQLite.

- HTML 5 Generated Reporting from SQL Lite->Modtable->SQLDriver, expose to Delphi & Albion custom SQL.

- Studied, Noted, Doubled Check, Surgical tweaked & Fix system integration components CAD/GIS/DB/Verb System, State bugs, little mistake all over, results state corruption and crashes.

- Improve More Efficient Verb System to update UI elements CAD/Grid/Attribute Editor incrementally. Join Tables - Incremental, not just rest all and refresh, compute CRUD left 3 ops X CRUD Right 3 ops become 9 datasets pairs results to forward on up or down the join relationships, super faster UI updates + Improve --+ Undo and Redo incremental and fast, not just reset and join rest all computation, replay undo and redo as new verb do operation, then nuke the last 2 verb operations chains off the stack.

Many further improvements to the system
- Enahnced dynamic fields
- look to enhance dataset alignment as typically you have 1 few that is aligned spatially to an index and another that is aligned to grid display view and one needs optimal indexes for both of these.

- Improvements to the Verb System, which still would like to apply in probably my own react/analguar frontend system one day or just generic sync tool.
Batching and aggregation of events into bundle optimize update events.