Revision History

19-Jul-2023

1.  Add session (cookie) support for Yahoo Finance.

22-Jun-2023

1.  Add logging support.

27-May-2023

1.  Update Yahoo quote to use the v10 service.  Refactored quotes section to remove csv structure since json has been in use for many years. 

03-Oct-2022

1.  Now using REQUESTS package to manage https sessions, rather than urllib.  HTTP requests are now managed by the library, including cookie management.  This change may be a "breaking change" for some users, since it requires the 3rd party requests package.  Requests is included in both ActiveState and Anaconda distrubutions, so it won't affect anyone using those, but basic Python installations (e.g., python.org) will need to add requests as described in Step 1 of the installation instructions.

04-Jan-2020

1.  Minor update to formally remove support for Google Finance quotes in sites.template and setup instructions.  Code sections for Google Finance were left intact for now, as a placeholder should functionality for Google or other (similar) service becomes available and/or necessary.

20-Nov-2019

1. Added support for importing manually downloaded ofx files, and applying "scrub" methods prior to sending to Money.

   • Setup creates an ./import subfolder (alongside ./xfr, etc). Running setup creates the folder, but you can add manually too.

   • Any ofx, qfx, etc. file can be dropped into folder. No need to rename.

   • When Getdata.py is run, it will process *all* files in ./import, and if it "looks like" an ofx file, it will be treated the same as a downloaded file (scrubbed, combined, etc).

   • If accepted as a valid ofx file, it is moved to the ./xfr folder (i.e., it is consumed).

   • An attempt is made to perform site-specific scrub routines on the ofx file prior to import.  This is done by matching the FID or BANKID value found in the ofx file vs those found in sites.dat site records.

   • General (not site-specific) scrub routines are always applied to download files, in an attempt to correct known incompatibilites with Money.

2. Added "delay" option to <Site> entries.   When defined, a delay will be added prior to sending a request to any account using that site.  The value of the delay parameter is in seconds, and defaults to 0.0 (i.e., no delay).

24-Aug-2018

• Added scrub routine for T.Rowe Price, to correct issues related to reinvested dividends and capital gains.

• Removed CLTCOOKIE field from request.  It is not supported by Quicken, or any version of Money after 2004.

28-May-2018

• When adding or updating an account in Setup, it will now query the institution, and give a list of valid accounts.  The user can still enter the account number manually, and will need to do so if/when a bank masks the account number in the reply (e.g., XXXX-XXXX-1234).

• When running GetData, if a connection fails for a given Site+Username combination, it will not attempt another connection with those same credentials at that site during the session.  The intent is to give a single error message, regardless of the number of accounts assigned to a given username at a site.  A new sites.dat option named skipFailedLogon can be used to override this behavior.

• Setup now sorts accounts by Sitename+Username+Account.

04-Apr-2018

• Replace invalid ampersand '&' chars with &amp; escape code.

24-Mar-2018

• Modified Yahoo quote to use longName parameter, when available, for company/fund name.

• 01-Jan-2018

• Update scrub routine to match recent changes at Discover Bank (again).

06-Dec-2017

• Patch to address change at Google Finance.

• Added GoogleURL option to sites.template, which can now be used in sites.dat.

10-Nov-2017

• New Yahoo Finance quote method, using their json service.

• Patch to address change in how Discover Bank encodes transaction FITID values.

27-Aug-2017

• Bug patch.

01-Aug-2017

• Added support for session tokens (cookies) in response to a recent change at Vanguard.

11-May-2017

• Minor edit to widen account number columns in Setup listing, due to yet another Discover change.

10-Apr-2017

• Bug patch.  New REINVEST scrub routine wasn't being called.

08-Apr-2017

• Updated scrub routine to handle malformed REINVEST transactions, where the sign (+/-) is passed incorrectly from the server.  Similar issue as previously addressed for transactions where BUY and SELL transactions use the wrong sign.

24-Mar-2017

• Updated https request to work with recent changes at Discover and others

• Added support for OFX 2.x (tested with ofxVer=211)

• Added support for user-specific clientUID pairs (by site url + username)

• Added skipZeroTransactions option to sites.dat configuration.  If Yes, remove $0.00 transactions from downloaded statements.

• Minor code updates and bug patches.

18-Aug-2016

• Updated Discover scrub routine to handle new format used by Discover Bank, vs Discover Card.

14-Sep-2015

• Bug fix when parsing Yahoo quote time format.

17-Feb-2015

• Added support for adding multiple instances of an account for a Site (bank).  This addresses the issue of providers like Fidelity Netbenefits, which allow multiple accounts to have the same account number (as odd as it sounds)  Each account uses a different user id for access, but the account numbers can be identical.  Two accounts for the same user (or couple) can therefore require the ability to download data for the same provider account, but match those to *different* accounts in Money.  Since the vendor-provided account number is used to Match accounts in Money, it presents a conflict.

To handle the condition, the account number assigned in Setup can now have a unique suffix, separated by a colon (e.g., 123456789:A).  When communicating with the provider, the base account number is used.  For the example, the script will request data for account 123456789 via the user id and password provided in account Setup.  The resulting statement will pass to Money with the full account number assigned (including the suffix), which can be uniquely matched to a Money account as-usual.  There is no limit to the number of times the base account number can be reused, so long as a colon (:) is used as the separator, and a unique suffix is used for each account entry.  It is recommended that the suffix be limited to 2-chars, so that Setup shows the full account on-screen, and to avoid any potential (but unknown) account length limitation in Money.

06-Sep-2014

• Modified file path handling to support running scripts on platforms other than Windows (platform independent).

25-Feb-2014

• Fix: Investment sell transactions mishandled by statement scrub routine.

• Fix: Exception related to forceQuotes when stock downloads are disabled.

14-Feb-2014

1. Bug fix: Yahoo time-out for non-US users, and error when YahooFinance is disabled.

25-Jan-2014

1. Added support for quotes from Google Finance.  The following options were added:

• EnableGoogleFinance :  Default=Yes  

• EnableYahooFinance : Default=Yes

Quote symbol lookup first occurs on Yahoo Finance, using their csv interface.  If this fails, and if EnableYahooScrape is enabled, then a second attempt is made to Yahoo by trying to parse the web interface that you see when looking up quotes on their site.  If this fails, and if EnableGoogleFinance is enabled, then a third attempt is made by looking on Google Finance.  If EnableYahooFinace is disabled, then the quote search goes straight to Google Finance on the first try.

If you have a stock symbol that is identical on both sites, then it will read from the first successful source.  If you have a symbol that only exists on Yahoo or Google, then it will only read from that source.

• • A summary of options related to quotes at this point is summarized below.  Note that the ForceQuotes option should only be used with non-US versions of Money.

09-Jan-2014

• Bug fix related to ForceQuotes option and the turn of the calendar year.

14-Nov-2013

• Added QuoteAccount option to sites.dat, enabling support for multiple quote currencies.

• Bug fixes for non-US versions of Money.

• Minor code edits and updates.

21-Sep-2013

• YahooScrape bug fix.

10-Sep-2013

• Changed default App Version to 2200.

• Added Yahoo quote support for European versions of Money

• Added INVTRANLIST tag set

• Added a new sites.dat option named forceQuotes (default = No).  Enabling this option will force a reconcile transaction in Money, which is required in some European versions to record a price change.  Enabling this option requires that the user accept an additional download statement each time Getdata is run, regardless of your CombineOFX setting. The option should only be enabled for European versions of Money, and only if there are problems downloading quotes.  Additionally, Money must be open before executing Getdata with ForceQuotes enabled.  If Money is closed, it may not update correctly, but it shouldn't hurt anything either.

• Updated YahooScrape code, which is called when the Yahoo quote API fails to return a valid quote response (if EnableYahooScrape is set to Yes).

14-Jul-2013

• Bug fix in _scrubDTSTART.

22-Mar-2013

• Bug fix in "time shift" scrub routine.

15-Mar-2013

• Added scrub routine for transaction sign issues (first noticed on Fidelity netbenefits).

• Added scrub routine to remove CORRECTACTION/CORRECTFITID sections (not supported by Money).

• Added sanity check for statements that contain <SEVERITY>Error (indicating an error requiring user review).

14-Jan-2013

• Added support for OFX 103.  Site entries allow a new parameter named ofxVer, and can have a value of 102 or 103 (Default=102).  A ClientUID value is automatically created, and saved to sites.dat as a static UID value.  

• Added a new sites.dat option named PromptInterval (Default=No).  If defined as "Yes", the user is prompted for the download interval (days) as an override to the global "default" and "site minimum" values defined in sites.dat.

• Added a new sites.dat option named CombineOFX (Default=No). If defined as "Yes", OFX statements are combined prior to sending to Money, resulting in a single “accept” for all statements inside Money.  It is highly recommended that this option not be enabled until *after* testing/verifying account settings one at a time.

• Added a new sites.dat option named quietScrub (Default=No).  If defined as “Yes”, statement scrubber messages are suppressed during download.

03-May-2011

• Write .\xfr\quotes.htm when collecting stock/fund quotes.  Defined the following user options in sites.dat:

• ShowQuoteHTM: Yes/No : Always show quotes.htm following download

• AskQuoteHTM: Yes/No : Ask to show quotes.htm following download (overrides ShowQuoteHTM)

• Setting both to “No” (default) will never display quotes.htm, but the file will still be written.

• Rather than showing the file every time, or asking each time, you can also open the quotes.htm file in your browser once and then copy/paste the link as a shortcut inside Money.  Dan has also indicated that he will use the link in his Money add-in.

• Enabled sanity check for <SECLIST> when a <INVPOS> exists.  Hopefully, this avoids the dreaded issue that *bombs* Money.

• Improved scrubber code:

• Recoded some routines for less complexity.

• Added routine for missing DTEND when a DTSTART exists.

• Scrubbers are more verbose, telling you what’s happening.

• Connection exceptions are handled more gracefully.

• Extended timeout for OFX connections to 30 secs.  This will create delays when a server is offline, but provides a more reliable service for slow (or highly active) servers.

• Added Yahoo "screen scraper" code for foreign FUNDS (provided by Cal Learner), after editing to fit code structure a little better.  A few notes:

  1. Added EnableYahooScrape: Yes/No option in sites.dat (default = yes).

  2. Grabs quotes for *funds* not supported by the native Yahoo csv interface.  Only executes if the Yahoo CSV-API fails.

  3. Screen scrapers may fail at any time (cross fingers).  There are a couple of sanity checks in place, so hopefully…trash doesn’t get to Money.

  4. The scraper does not currently support “last price” or “change in price”.  This feature is used by the new quotes.htm writer, but won’t affect Money import operations.

• Added timeOffset option for site entries in sites.dat.

• timeOffset: hrs :   Add hrs to statement DTASOF field(s).  Default = 0 (zero).  Negative hrs will adjust DTASOF time backwards.

• Added QuoteCurrency: USD option to sites.dat.  Defines currency for quotes.  Default = USD.

• Use ticker symbol for quote name when a *null* name is received, but only if a quote value other than 0.00 is provided by Yahoo.

• The "latest date/time" for ticker quotes is now used for the statement date (DTASOF field).  Until now, ~ "today's date" has been used.  Example, if there are 10 stock/fund quotes retrieved, but the latest date/time stamp for any of those is "15-Feb-2011 @ 12:00", then that time stamp is used for the "statement date" (which is used by Money when setting the quote date/time for them all).

• Added slight delay when sending results to Money (between file submits).

• Added 10 sec timeout for Yahoo quote downloads.

• Added options for Yahoo quote symbols in sites.dat:

  1. m:value:  Currency multiplier option. Quote values will be adjusted (multiplied) by this (optional) value during download.

  2. s:value: User defined symbol to send to Money (rather than Yahoo ticker symbol).  

     • Example:  BP.L  m:0.01  s:MyBP

     • Multiply quotes received for BP.L by 0.01 (i.e, convert from pence to pounds), before sending the results to Money.

     • Pass quote to Money with symbol=MyBP (rather than BP.L).  The Yahoo quote will be for ticker = BP.L, but Money will see it as ticker=MyBP.

• Updated the sites.template file to include placeholders for all new options described above.

13-Oct-2010

• Ampersand (&) caused errors when used in a SiteName entry in sites.dat.  Fixed.

• Date format in QuoteHistory.csv fixed.

19-Sep-2010

• If a connection fails, gracefully continue rather than aborting execution.  Notify user afterwards.

• Added option to save quote history to QuoteHistory.csv file.  This option is enabled by adding a flag to the sites.dat file:  SaveQuoteHistory: Yes.  If not defined, the default is No.

• Added support for optional Yahoo! Finance site(s).  This option is enabled by adding a flag to the sites.dat file:  YahooURL: http://ca.finance.yahoo.com.  If not defined, the default value is http://finance.yahoo.com.

• Added support for spaces and illegal Windows file characters to sitename fields in sites.dat.

• Replace null time values with a 12:00:00 value (i.e., assume noon time if no time is given).  This prevents Money from pushing a date back a day (due to timezone differences).

• Fixed all reported bugs/user issues to date.

10-May-2010

• SaveTickersFirst = Yes flag (in sites.dat) now forces download of quotes first so that the

  • "file created timestamp" is the earliest.  Money apparently does not always use a FIFO method for importing files in an input queue, but appears to use the file timestamp. Thanks to Ray for finding and verifying this issue.

• Fixed a bug that only allowed testing/deleting accounts numbered 1 to 9.  An unlimited number of accounts is now supported.

• If a connection fails, it is now reported and the exit occurs more gracefully. Getdata.py will now move on to the next account or quote server and try again.

18-Mar-2010

• Added "interactive" mode for Getdata.py.

• Added SaveTickersFirst user-configuration option to sites.dat.  The default is to send statement results to Money first, and then stock/fund quotes.  Defining this parameter with a value of "Y" or "Yes" will send stock/fund quotes first, followed by account statements.  Note that stock quotes are always downloaded from the internet last, but enabling this option will send the results to Money first.

• Skip stock/fund symbols that aren't recognized by Yahoo! Finance (rather than throw an error)

• Minor code updates.

08-Mar-2010

• Modified Getdata.py to download stock/fund quotes even if no accounts (i.e., statement downloads) have been defined.  The user can simply run Setup.py and enable stock/fund downloads and then edit the stock/fund entries in sites.dat.

04-Mar-2010

The scripts seem to be reaching a point of being "stable" for general use, but I do expect to add a few more "features".  It won't surprise me at all to hear about missed "bugs" either, so please let me know if you find one. 

Issues addressed:

• Fixed a significant bug in the parser for reading the sites.dat file.  Previously, it was possible for part of a site URL or other field to have text matching a field name.  The new parsing method matches entries with known field names using a different method.  FIXED.

• Minor typos and enhancements to Setup.py

• Setup.py

  1. Added two new BANK types.  Allowable bank types now include: Checking, Savings, Money Market, and Credit Line

  2. Now lists account type for all accounts, not just the BANK types.

  3. Corrected bug related to entering BANK account type.