Scalable Parallel Varfiler

Scalable Parallel VarFiler version 1.85

Authors: Amine Moulay Ramdane

Scalable Parallel Variable filer and streamer for Delphi and Freepascal that can be used also as a parallel hashtable and that uses ParallelHashList (Parallel Hashtable) with O(1) best case time complexity, and when there is collisions in the hashtable , the average time complexity will be O(log(n)) in each bucket were the collisions happened. The parallel Hahstable uses lock striping and my scalable RWLock in each bucket of the parallel hashtable , this allows multiple threads to write and read concurently. also ParallelHashList maintains an independant counter , that counts the number of entries , for each segment of the hashtable and uses a lock for each counter, this is also for better scalability.

Description:

Scalable ParallelVarFiler is a Parallel HashTable that can be saved automatically or manually to a file or to a stream or to a string and it can be restored from a file or from a stream or from a string back to the Hashtable in memory, and it's fault tolerant to power failures etc.

You can collect different types of variables and save them into one file or stream. TParallelVarFiler reads and writes on files, streams and Strings. You can collect Strings, Integers and Dates, in fact everything that can be stored in a variant. In addition you can collect Streams.

You can use also ParallelVarFiler to send parameters over network. It can be used for example to share data by any IPC mechanism.

And please look at test.pas and test1.pas a parallel variable filer examples - compile and execute them...

Now you can use ParallelVarFiler as a small to medium database, i have added a Clean() and DeletedItems() methods and i have changed the constructor, now you can pass a file name to the constructor where the data will be wrote and updated and deleted... if you pass and empty file name the data will be updated and wrote and deleted only in memory. When you use a file name in the constructor, many readers and one writer can proceed concurrently. But when you pass an empty file name to the constructor, many writers and many readers can proceed concurrently in memory. and when you read a data item it only use the parallel hashtable in memory, hardisk is not used.

And when you pass a file name to the constructor and delete or update the items, those items will be marked deleted from the file, and you can use DeletedItems() to see how many data items are marked deleted in the file, and use after that the Clean() method to delete completly those data items from the file.

How can you store mutiple data inside a database row and map it to a key ?

You can simply use Parallel Variable Filer for that and stream your mutiple data variants and map them to a key...

As you know ParallelVarFiler is using ParallelHashList (a parallel hashtable that scales on multicores), so when you are using the GetKeys() method , and you want to get the data of those keys don't forget to test if the variant is not empty by using the VarIsEmpty() function, and when you are using GetStream() you can test if the data exists by testing the return boolean value from GetStream().

ParallelVarFiler is easy to learn, i have documented all the methods , and please read inside ParallelVarFiler.pas about

them.

Now ParallelVarFiler is Fault tolerant to power failures etc. i have done a simulation of power failures and data file damages and ParallelVarFiler is recovering from power failures and damages of the data file ...

If AnalyzeVarfiler() returns ctCorrupt if there is a power failure or something like that that gives a corruption, you have to fix the format of the archive with FixVarfiler() that will fix the format of the archive and recover from a power failure corruption etc.

Please see the test.pas example inside the zip file to see how i am using it...

But please read the following:

This software is provided on an "as-is" basis, with no warranties, express or implied. The entire risk and liability of using it is yours. Any damages resulting from the use or misuse of this software will be the responsibility of the user.

please look at the test.pas and test1.pas example inside the zip file to see how to use ParallelVarFiler...

Here is the methods that have been implemented:

PUBLIC METHODS:

constructor Create(file1:string,size,mrews:integer;casesensitive:boolean);

- Creates a new VarFiler ready to use, with size and with mrews number of mrews and casesensitive for case sensitive keys,the number of MREWS(multiple-readers-exclusive-writer) must be less or equal to the Hashtable size and file1 is the the file to write to, if file1 is an empty string the data will be written only to memory and use SaveToFile() to write to a file. The returned exceptions of the constructor is ELoadIndex if it can not load the index or EUnknownFileFormat if it isn't the right varfiler format.

destructor Destroy;

- Destroys the ParallelVarFiler object and cleans up.

procedure Clear;

- Deletes all contents.

function Clean:boolean

- Cleans the deleted items from the file.

function DeletedItems:integer

- Returns the number of items marked deleted.

function LoadData:boolean

- Loads the data from the file passed to the constructor.

function Delete(Name : String):boolean;

- Deletes the variable with Name.

function Exists(Name : String) : Boolean;

- Returns True if a variable with Name exists

procedure GetKeys(Strings : TstringList);

- Fills up a TStringList descendant with all the keys.

function Count : Integer;

- Returns the number of variables

function Add(Name : String; Value : Variant ):boolean;

- Adds a new variable , given the Name

function AddStream( Name : String;Stream : TStream ):boolean;

- Adds a new Stream, given the Name

function Update(Name : String; Value : Variant):boolean;

- Updates the new variable , given the Name

function UpdateStream(Name : String; Stream : TStream ):boolean;

- Updates a new Stream , given the Name

function GetStream(Name : String;Stream : TStream):boolean;

- Fills up the stream with data

procedure SaveToStream(Stream : TStream);

- Saves all variables, streams and Graphics to a stream.

procedure SaveToFile(FileName : String);

- Saves all variables, streams and Graphics to a file.

procedure SaveToString(out S : String);

- Saves all variables, streams and Graphics to a string.

procedure SaveToStreamAndDelete(Stream : TStream);

- Saves all variables, streams and Graphics to a stream

and delete after that all the data inside the hashtable

and inside the file.

procedure SaveToFileAndDelete(FileName : String);

- Saves all variables, streams and Graphics to a file.

and delete after that all the data inside the hashtable

and inside the file.

procedure SaveToStringAndDelete(out S : String);

- Saves all variables, streams and Graphics to a string.

and delete after that all the data inside the hashtable

and inside the file.

function LoadFromStream(Stream : TStream):boolean;

- Loads all variables, streams and Graphics from a stream.

function LoadFromFile(FileName : String):boolean;

- Loads all variables, streams and Graphics from a File.

function LoadFromString(S : String):boolean;

- Loads all variables, streams and Graphics from a string.

function VariantTypeName(const AValue: Variant): String;

- Returns the type name of a variant.

function IsVarFilerFromStream(Stream : TStream):boolean;

- Returns true if the Stream is in a Parallel VarFiler format, and false if not.

function IsVarFilerFromFile(FileName : String):boolean;

- Returns true if the File is in a Parallel VarFiler format, and false if not.

function IsVarFilerFromString(S : String):boolean;

- Returns true if S is in a Parallel VarFiler format, and false if not.

function FixVarfiler(Stream:TStream):boolean;overload;

- Fix the format of the Stream

function FixVarfiler(filename:string):boolean;overload;

- Fix the format of the file

function AnalyzeVarfiler(filename:string):TypeError;overload;

- Analyze the format of Archive file, the returned TypeError is ctCorrupt if the format of the file is corrupt or ctUnknown if the format of the file is unknown.

function AnalyzeVarfiler(Stream:TStream):TypeError;overload;

- Analyze the format of the Archive Stream, the returned TypeError is ctCorrupt if the format of the stream is corrupt or ctUnknown if the format of the stream is unknown.

function Stream2String(Stream1: TStream;var Mystring:string): boolean;

procedure String2Stream(var String2BeSaved:string; Stream1: TStream);

procedure Variant2Stream(var VR:variant; Stream1: TStream);

function Stream2Variant(Stream1: TStream;var VR:variant): boolean;

PUBLIC PROPERTIES:

Items : Variant

- Gets the value (indexed).

Language: FPC Pascal v2.2.0+ / Delphi 7 to Delphi 2007: http://www.freepascal.org/

Operating Systems: Windows, Mac OSX , Linux...

Required FPC switches: -O3 -Sd -dFPC -dFreePascal

-Sd for delphi mode....

Required Delphi switches: -$H+ -DDelphi

For Delphi XE-XE7 use the -DXE switch

{$DEFINE CPU32} and {$DEFINE Windows32} for 32 bit systems

{$DEFINE CPU64} and {$DEFINE Windows64} for 64 bit systems