Metatrader Python Integration

This is my solution for the problem of mql4 programs not being sufficiently able to communicate with the outside world. It adds the ability to run arbitrary python code (even multi-threaded) directly from within an mql4 script, expert advisor or indicator. Possible use cases include:
  • import the RPy bindings and then directly call R functions from mql4
  • import thread and socket or BaseHTTPServer or twisted and run network servers or clients
  • use the above to remote control metatrader via web interface
  • import one of the many database drivers/wrappers and directly access databases
  • remote control other applications via their COM interface
  • ...
  • import antigravity and just fly away!

This is still work in progress, it is far from complete, expect the API to change at any moment!

For the impatient: Scroll down until the end and find the attached files. You will only need the two files py26.dll and py26.mqh. Copy the .dll into experts\libraries and the .mqh into experts\include. Naturally you will of course also need to have Python 2.6 (http://www.python.org/) installed on your machine.

Take a look into py26.mqh and make sure you don't have a version where i accidently forgot to change back the path in the #import statement form #import "c:\somewhere\on\bernds\notebook\py26.dll" to #import "py26.dll". This line must read #import "py26.dll" so metatrader will look for it in the libraries folder.
 
The .rar file is not needed for using this, it only contains the source code of the dll for those of you that want to know what is inside, want to compile it themselves or improve it.

The dll is written in Free Pascal, the free successor of TurboPascal and Delphi, using the Lazarus IDE. People keep asking me why on earth i use Pascal and not C/C++, the complete answer and a new question to those asking me is here. However, you dont need to know anything about Pascal or C or ever need to install or use them to make use of this library, just be assured that this dll is written in a proper language.

To save myself some time repeating the documentation for all the API functions i just copied the contents of the mql4 include file below.

As you can see when looking over the API this is basically a small subset of the complete python API, my dll provides all these functions more or less exactly like they are defined by python.h and you may ask: why the need for a wrapper dll at all, why not just import the API functions directly in mql4? The answer is simple: We need the Python interpreter to run in it's own thread and mql4 has no support for threading, furthermore mql4 can only import functions that were exported using the __stdcall calling convention, the python api exports its functions with __cdecl which means the stack pointer is handled differently, trying to directly import functions from python26.dll in mql would immediately crash the whole application with access volations, the only way around this is to write a thin wrapper dll that starts a new thread for the interpreter, imports the functions from python with __cdecl, wraps them into PyGILState_*() calls to make them thread safe and exports everything to mql4 with __stdcall convention. Pointers to Python objects are simply cast to integers, mql can not dereference these pointers but it also doesnt need to do so, instead it can simply treat them like handles and pass them back to the dll which will cast them back into pointers to use them. (I know this is not the textbook example of portability but it doesnt need to be portble, it just needs to work on x86/win32).


//+------------------------------------------------------------------+
//|                                                         py26.mqh |
//|                                                     Bernd Kreuss |
//|                                             mailto:7ibt@arcor.de |
//|                                                                  |
//|                                                                  |
//|               Python Integration For Metatrader 4                |
//|                                                                  |
//+------------------------------------------------------------------+
#property copyright "Bernd Kreuss"
#property link      "mailto:7ibt@arcor.de"

#import "py26.dll" // version 1.3.0.x

/**
* Initialize the Python environmant. This will start a new
* thread which will initialize Python and enter a sleep() loop.
* Dont call this directly, call PyInit() instead. (see below)
*/
void PyInitialize();

/**
* Return True if PyInitialize() has already been called
*/
bool PyIsInitialized();

/**
* Decrease the reference counter of a Python object by one and free the
* object if the counter reaches zero. You may only call this for objects
* that you OWN yourself, not for BORROWED references.
*/
void PyDecRef(int p_object);

/**
* Increase the reference counter by one
*/
void PyIncRef(int p_object);

/**
* Execute an arbitrary piece of python code
*/
void PyExecute(string python_source);

/**
* Evaluate a python expression and return your NEW OWN
* reference to the result. For example if foo.bar.baz is a list
* and you want a reference to it so you can directy loop through
* its members then just call PyEvaluate("foo.bar.baz") and use
* the return value as a handle for the PyLookupList() function.
* You can also use this handle for PyListAppend() because it is
* *not* a copy but rather a reference to the original list.
*
* After you are done with using the handle returned by PyEvaluate()
* you MUST call PyDecRef() to signal Python that you are now done
* with this object, this will restore the reference counter to
* the value it had before or free it completely if the object
* was created by the code you evaled (return value of a function
* or the result of a calculation which python will no longer
* need after you read it)
*/
int PyEvaluate(string python_source); // new reference, PyDecRef() after using!

/**
* return a BORROWED reference to the __main__ dict
*/
int PyMainDict();

/**
* return a BORROWED reference to the item in the dict
* specified by name.
*/
int PyLookupDict(int p_dict, string name);

/**
* return a BORROWED reference to the item in the list
* specified by index.
*/
int PyLookupList(int p_list, int index);

/**
* return the value of the object as int
* (if it is a numeric object)
*/
int PyGetInt(int ptr_int);

/**
* return the value of the object as a double
* (if it is a numeric object)
*/
double PyGetDouble(int p_double);

/**
* return the value of the object as a string
* (only if it actually is a string object)
*/
string PyGetString(int p_string);

/**
* create a new integer obect. You will OWN a reference,
* so take care of the reference counter
*/ 
int PyNewInt(int value);

/**
* create a new double (actually a python float) object.
* You will OWN a reference, so take care of the reference counter
*/ 
int PyNewDouble(double value);

/**
* create a new string obect. You will OWN a reference,
* so take care of the reference counter
*/ 
int PyNewString(string value);

/**
* append the item to the list. This function will NOT steal
* the reference, it will create its own, so ownership
* will not change, so if it was your OWN and NOT a
* borrowed reference you must decref after you are done
*/
void PyListAppend(int p_list, int p_item);

/**
* return the size of the list object
*/
int PyListSize(int p_list);

#import

/**
* below are some higher level abstractions, here you dont have to
* care about reference counting when using the functions, you will
* not be exposed to handles of python objects
*/


/**
* initialize the Python environment. This should be called
* from your init() function. It is safe to call it a second
* time, subsequent calls will just be ignored.
*/
void PyInit(){
   if (!PyIsInitialized()){
      PyInitialize();
      PyExecute("import os, sys");
      PyExecute("os.chdir(\""+TerminalPath()+"/experts\")");
      PyExecute("sys.path.insert(0, os.getcwd())");
   }
}

/**
* Evaluate a python expression that will evaluate to an integer
* and return its value
*/
int PyEvalInt(string python_source){
   int p_res = PyEvaluate(python_source);
   int res = PyGetInt(p_res);
   PyDecRef(p_res);
   return(res);
}

/**
* Evaluate a python expression that will evaluate to a double
* and return its value
*/
double PyEvalDouble(string python_source){
   int p_res = PyEvaluate(python_source);
   double res = PyGetDouble(p_res);
   PyDecRef(p_res);
   return(res);
}

/**
* Evaluate a python expression that will evaluate to a string
* and return its value
*/
string PyEvalString(string python_source){
   int p_res = PyEvaluate(python_source);
   string res = PyGetString(p_res);
   PyDecRef(p_res);
   return(res);
}

/**
* append the array of int to the python list given by its name.
* the list must already exist. The same could be achieved
* by putting PyExecute() calls with generated python code
* into a loop but this would invoke parser and compiler for
* every new list item, directly accessing the python objects
* like it is done here is far more effective.
*/
int PyListAppendInt(string list_name, int array[]){
   int list,item,len,i;
   list = PyEvaluate(list_name);
   len = ArraySize(array);
   for (i=0; i<len; i++){
      item = PyNewInt(array[i]);
      PyListAppend(list, item);
      PyDecRef(item);
   }
   len = PyListSize(list);
   PyDecRef(list);
   return(len);
}

/**
* append the array of double to the python list given by its name.
* the list must already exist.
*/
int PyListAppendDouble(string list_name, double array[]){
   int list,item,len,i;
   list = PyEvaluate(list_name);
   len = ArraySize(array);
   for (i=0; i<len; i++){
      item = PyNewDouble(array[i]);
      PyListAppend(list, item);
      PyDecRef(item);
   }
   len = PyListSize(list);
   PyDecRef(list);
   return(len);
}

/**
* append the array of string to the python list given by its name.
* the list must already exist.
*/
int PyListAppendString(string list_name, string array[]){
   int list,item,len,i;
   list = PyEvaluate(list_name);
   len = ArraySize(array);
   for (i=0; i<len; i++){
      item = PyNewString(array[i]);
      PyListAppend(list, item);
      PyDecRef(item);
   }
   len = PyListSize(list);
   PyDecRef(list);
   return(len);
}



/* Some words:
 *************


* One Interpreter
  ===============

  All expert advisors and indicators share the same
Python interpreter with the same global namespace, so you should
separate them by encapsulating all in classes and instantiate and
store them with all their state in variables named after the symbol
(or maybe even symbol + timeframe).


* Init
  ====
 
  Put your Python classes into Python modules, the import path
is <metatrader>\experts, the same folder where your EA's mql code
is located, so a simple PyExecute("import yourmodule"); in your
init() will import the file yourmodule.py from this folder. Then
instantiate an instance of your main class with something like
PyExecute(Symbol() + Period() + " = yourmodule.yourclass()");
This way each instance of your EA can keep track of its own
Python counterpart by accessing it via this global variable.

  Your init() function may look similar to this:
 
int init(){
   // initialize Python
   PyInit();
  
   // import my module
   PyExecute("import mymodule");
  
   // instantiate some objects
   PyExecute("myFoo_" + Symbol() + Period() + " = mymodule.Foo()");
   PyExecute("myBar_" + Symbol() + Period() + " = mymodule.Bar()");
  
   return(0);
}


* Deinit
  ======
 
  Use the deinit() function of the EA or Indicator to destroy
these instances, be sure to terminate all threads they may have
started, make sure you can terminate them fast within less than a
second because Metatrader has a timeout here, wait inside python
in a tight loop with time.sleep() until they are terminated before
returning!

  Your deinit() function may look like this:
 
int deinit(){
   // tell my objects they should commit suicide by
   // calling their self destruction method
   PyExecute("myFoo_" + Symbol() + Period() + ".stopAndDestroy()");
   PyExecute("myBar_" + Symbol() + Period() + ".stopAndDestroy()");

   return(0);
}


* Global unload hook
  ==================
 
  If the last EA that used Python has been removed the Python
interpreter itself will be terminated and unloaded.
 
  You can register cleanup functions (do it per imported module, not
per instance!) with the atexit module, it will be called after the
last EA's deinit(), again as above make it wait for all cleaning
action to be finished before returning, these are the last clock
cycles that will be spent inside Python because at this time there
is only one system thread left and if this function returns the
python interpreter will be frozen and then immediately unloaded.

*/




Subpages (1): The Thread Problem
Č
ċ
py26-dll-source.rar
(3k)
Bernd K.,
Aug 29, 2009, 11:20 AM
ċ
py26.dll
(144k)
Bernd K.,
Aug 26, 2009, 10:54 AM
ċ
py26.mqh
(10k)
Bernd K.,
Oct 26, 2009, 8:21 AM
Comments