dvr-recover

Description (Readme)

This section may contain excerpts from files of the current development tree. The mentioned information might not match the behaviour of the current stable release completely.

      ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
      ::      _                                                         ::
      ::   __| |_   __ _ __      _ __  ___   ___  _____   __ ___  _ __  ::
      ::  / _` \ \ / /| '__|____| '__|/ _ \ / __|/ _ \ \ / // _ \| '__| ::
      :: | (_| |\ V / | | |_____| |  |  __/| (__| (_) \ V /|  __/| |    ::
      ::  \__,_| \_/  |_|       |_|   \___| \___|\___/ \_/  \___||_|    ::
      ::                                                                ::
      ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::



 Description
~~~~~~~~~~~~~

 dvr-recover is an incredible Python-powered project helping you to recover
 recordings of digital video recorders. In other words: The script is able to
 find video files on the surface of the device's hard disk drive.

 Practically, the process is much more complex. Have a look at the following
 sections if you want to gain more knowledge about the details.

 There are many devices on the market, not all models can be supported. See the
 list of tested devices to see if your device has been already tested.



 License
~~~~~~~~~

 Copyright (C) 2010 Stefan Haller <haliner AT googlemail DOT com>

 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation, either version 3 of the License, or
 (at your option) any later version.

 This program is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.

 You should have received a copy of the GNU General Public License
 along with this program.  If not, see <http://www.gnu.org/licenses/>.



 Tested Devices
~~~~~~~~~~~~~~~~

 Currently only Panasonic DMR devices have been tested. If you successfully
 recovered the content of your drive (completely or partly), please drop me a
 mail!

 Working devices:

  * Panasonic DMR-EH55
  * Panasonic DMR-EH56
  * Panasonic DMR-EH57
  * Panasonic DMR-EX77
  * Panasonic DMR-EX85
  * Panasonic DMR-XW300
  * Panasonic DVM-E80H



 System Requirements
~~~~~~~~~~~~~~~~~~~~~

 To run the script you need a Python interpreter (2.x, tested with 2.5 and
 higher).



 Usage
~~~~~~~

 If you execute the script without any parameters, the script will print the
 usage message where all parameters, switches and settings are explained in
 detail.

 Executing the script is very easy: Simply open a terminal (on Windows:
 [Win] + [R] -> enter "cmd" and press enter). At first you have to switch to the
 directory where the script is located and then you can fire up the Python
 interpreter.

 Linux:

   cd /path/to/script/
   python dvr-recover.py [parameters]

 Windows:

   cd C:\Path\to\script\
   python dvr-recover.py [parameters]



 Guide
~~~~~~~~

 This section will guide you through the complete process of recovering the
 data from you hard disk drive.

 Naturally the hard disk drive of the digital video recorder must be plugged
 into the computer. So you need to open the device and fetch the hdd. You can
 either attach the disk directly to the IDE ports of your motherboard or you
 can use a IDE-USB-bridge.

 It's recommended that you create an image of your disk, but if you think you're
 lucky you can even use the hard disk drive directly as input on Linux. Note
 that direct access to the hard disk drive on Windows is currently not supported
 by dvr-recover. On Windows you have to use 3rd party programs to create the
 image. On Linux you can use the standard tools (be sure that you have root
 permissions):

   (Assuming that your disc is labeled /dev/sdb)
   dd if=/dev/sdb of=disk.img bs=10M

 This guide assumes that you have an exact bitwise copy of the hdd on your
 file system. Note that it's also possible to split your image into smaller
 parts. The script is able to handle this situation correctly. But you have to
 be sure that you specify all parts in the correct order in the later process.
 If you want to use the hdd directly, you will need to substitute the filenames
 in this guide with the device identifier (/dev/sdb, ... on Linux).

 The script will automatically create a sqlite3 database to store all
 settings and gained information. If somethings goes heavily wrong, you will
 be able to reset the state of the program by deleting the database. To do
 so delete the file "dvr-recover.sqlite" in the working directory of the
 script.

 Before you the script can analyze your hard disk drive, you have to go through
 a initial setup first. The script needs at least the information about the
 input filename(s). But there are many other settings to tweak -- most of them
 have sensible default values. For more information read the usage message of
 the script by passing "usage" as parameter.

 If you call the script with passing "setup" as parameter, it will print the
 current values of all settings:

   python dvr-recover.py setup
|  input_file: /home/user/disk.img.0001
|  [...]
|  input_file: /home/user/disk.img.9999
|  export_dir: /tmp/export
|  blocksize: 2048
|  min_chunk_size: 25600
|  max_create_gap: 90000
|  max_sort_gap: 90000

 To change a setting you have to pass the "setup" parameter and additionally the
 setting to change with the new value:

   python dvr-recover.py setup blocksize 2048

 The only exception is the handling of input files:

   python dvr-recover.py setup input clear
   python dvr-recover.py setup input add disk.img.0001
   python dvr-recover.py setup input add disk.img.0002
   python dvr-recover.py setup input del disk.img.0001
   [and so on...]

 Use the parameter "clear" to clear the list of input files. Also available
 are the parameters "add" and "del" -- both take a filename as argument.

 If you want to reset all settings, you have to use the parameter "reset":

   python dvr-recover.py setup reset

 Now you should be able to start the main feature of the script: recovering
 you hard disk drive.

 At first let the script analyze your hard disk drive. Call the script
 with passing the parameter "create". This process will take quite a long
 time to complete. You can interrupt the process at any point by pressing
 [CTRL] + [C]. On the next call with the parameter "create" the script will
 automatically resume the process where it was interrupted.

   python dvr-recover.py create
|  [ 29.5%] 297457/1006929 blocks (9915.2 bl/s; 19.4 MiB/s): 5 chunks
|  [ 65.0%] 654794/1006929 blocks (11911.2 bl/s; 23.3 MiB/s): 6 chunks
|  [ 91.3%] 918841/1006929 blocks (8786.1 bl/s; 17.2 MiB/s): 6 chunks
|
|  Finished.
|  Read 1006929 of 1006929 blocks.
|  Found 7 chunks.
|  Took 100.97 seconds.
|  Average speed was 9972.6 blocks/s (19.5 MiB/s).

 (The file used for generating the output above was very very small.)

 After finishing the analyze of the hdd the script will forbid all further
 calls with the parameter "create", because this would cause data loss
 (all data of the last analyze would be reseted first). You must explicitly
 reset the information in the database by passing the parameter "clear":

   python dvr-recover.py clear

 (THIS WILL DELETE ALL GATHERED DATA FROM THE DATABASE!)

 dvr-recover has the ability to sort the chunks and find chunks of the same
 recording. To start this process pass the paremter "sort" to the script:

   python dvr-recover.py sort

 To revert the effect of "sort", pass "reset".

   python dvr-recover.py reset

 At every stage you can display the chunk info by passing "show".

   python dvr-recover.py show
|  ----+--------------+--------------+--------------+--------------+------------
|      |  Block Start |   Block Size |  Clock Start |    Clock End | Concatenate
|  ----+--------------+--------------+--------------+--------------+------------
|    1 |       391379 |       615548 |    250031969 |    402836926 |     False
|    2 |       223188 |       168189 |    401478251 |    484558534 |     False
|    3 |       107989 |       115197 |    546892241 |    592303341 |     False
|    4 |        80992 |        26995 |   1085209725 |   1100156779 |     False
|    # |        53995 |        26995 |   1100157071 |   1115282776 |      True
|    # |        26998 |        26995 |   1115283069 |   1129877061 |      True
|    # |            0 |        26996 |   1129877208 |   1144930337 |      True

 In the first column you can see the index of the chunk. A '#' is printed if the
 chunk is concatenated to the previous one.

 The last thing is exporting the recordings. If the scrit has found chunks
 of the same recording, they will be automatically concatenated. You can
 either export all recordings in the database or only a single one.

   python dvr-recover.py export

 This will export all recordings. You can also specify a chunk id to export. In
 this example the script would export only the chunk with the id 4 (including
 the chunks which should be concatenated to this one).

   python dvr-recover.py export 4

 All the basics are explained -- you should be able to use the script now. Of
 course, there are a lot of other things that were not mentioned in this guide.
 For more details have a look at the usage message or the source code.



 Tuning the parameters
~~~~~~~~~~~~~~~~~~~~~~~

 Is there any possibility to tune the script and recover more files? Yes, it is!

 You can tune the process of chunk detection with parameters. The most important
 parameter is "min_chunk_size". The smaller the value is, the more the script
 finds. But the problem is that the possibility to find "garbage" is even
 higher.

 The default setting is that the program ignores chunks smaller than 50 MB
 (25600 blocks). If you set it to 0, the script will find everything, even if
 it's only one frame of a recording. If you play a bit with this parameter you
 maybe will recover more recordings completely.

 IMPORTANT: After changing the parameters run the complete scan again (parameter
            "create")!

 To change the setting run:

   python dvr-recover.py setup min_chunk_size [NEW_VALUE]

 [NEW_VALUE] must be given in blocks.
 Blocks * blocks size = size in byte. The default block size is 2048 byte.

 I tried to set the default value to a sensible default. If somebody can prove
 that a smaller value is better, I'll set the default value for later releases
 to a lower value.



 Homepage and Contact
~~~~~~~~~~~~~~~~~~~~~~

 Stefan Haller <haliner AT googlemail DOT com>

 http://sites.google.com/site/haliner/dvr-recover

 Feel free to contact me if you have any additional suggestions or you've
 found a bug. Feedback is always welcome! IF YOU THINK THE SCRIPT IS TERRIFIC,
 I WOULD BE VERY GLAD IF YOU LET ME KNOW THAT.

Installation

This section may contain excerpts from files of the current development tree. The mentioned information might not match the behaviour of the current stable release completely.

There is no need to install dvr-recover. Simply extract the tarball or the
zipfile anywhere and execute the script.

Consult the Readme for more information.

Usage

This section may contain excerpts from files of the current development tree. The mentioned information might not match the behaviour of the current stable release completely.

dvr-recover - extract MPEG2 files of digital video recorder hdd
===============================================================

Version: 0.6


Copyright (C) 2010 Stefan Haller <haliner AT googlemail DOT com>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.


Sqlite database file
--------------------

The script will create a sqlite3 database in the current directory with the
name "dvr-recover.sqlite". All settings and data will be stored in this
database.


Setup
-----

Before you can use dvr-recover you will have to change the default settings.
You can do this by simply calling the script with the parameter "setup"
followed by the setting to change and the new value. To change the value of
the blocksize setting call:

    python dvr-recover.py setup blocksize 2048

Here is a full listing of all available settings:

   The square brackets in this listing are representing the type of the value.
   Replace them with an integer for "[integer]". Don't copy this square
   brackets, so write

    blocksize 2048

  instead of

    blocksize [2048]


  input clear            This option sets the path of the hard disk drive
  input add [filename]   file used as input. You can either use a copy of the
  input del [filename]   block device (created with something like dd or so) or
                         the block device directly (required root privileges).
                         The file must be readable. It's possible to specify
                         multiple hdd-files by calling the add parameter
                         multiply times. The script will threat the single files
                         as one big file. That way you can split the hdd into
                         smaller pieces.

                         parameter clear:   clear list of input file
                         parameter add:     add one file to list of input files
                         parameter del:     delete one file from list of input
                                            files

  export_dir [string]    Defines where the output should be written to. Must
                         match an existing path. Both relative and absolute
                         paths are accepted. Current directory is "./".

  blocksize [integer]    The blocksize of the filesystem can be configured
                         with this option. Set this to a value in bytes.
                         The default value is 2048 bytes. Probably this value
                         should work, but if not, you're free to tune it.

  min_chunk_size [integer]
                         If the script finds chunks smaller than this size
                         (value must be given in blocks!), it will ignore them
                         silently. If this value is too small, the script
                         will find chunks that were deleted or can't be used.
                         Otherwise, if the value is too big, valuable chunks
                         will be ignored. The default value is 25600 blocks
                         (50 MiB by blocksize of 2048 bytes).

  max_create_gap [integer]
                         The script will split the stream into two chunks if
                         it finds two frames where the timecode differs more
                         than this value. MPEG uses a clock of 90 kHz.
                         So the default value of 90,000 ticks equals one second.

  max_sort_gap [integer]
                         See maxcreategap. This value is used to concatenate
                         two chunks if the difference of the timecode is smaller
                         than this value. The default value of 90,000 ticks
                         equals one second.


Input hdd file
--------------

You can create either a copy of the hdd or use the hdd directly as input.

Linux:
  If you want to copy the hdd (assuming the hdd is /dev/sdb) then you can use
  this dd command: (setting the blocksize to 10MB should increase performance.)

    dd if=/dev/sdb if=hddfile bs=10MB

  If you want to use the hdd directly as input, use this as value for hdd-file:
  (Assuming that the hdd is /dev/sdb.)

    hdd-file=/dev/sdb

Windows:
  Use a tool to create a copy of the hdd. Direct access to Windows device
  files (\\.\PhysicalDrive0, ...) is not supported at the moment.


Steps:
------

Step 1: Export information of chunks
  The hdd file will be analyzed and all necessary information are collected.
  This step may take quite a long time, be patient! This step is only necessary
  once. All other steps will use the stored chunk info to save time.
  (The script tries to find mpeg headers and extract the timecode. Depending on
  the timecode it's possible to split the stream into separate chunks.)

  Parameter: create

Step 2: Analyze and sort chunks
  This step will analyze the stored chunk info and sort the chunks. The tools
  tries to find parts of the same recording (by analyzing the timecode
  informationof the chunks) and bring them into the right order.

  Parameter: sort

Step 3: Show chunks
  You can list all chunks to make sure that the program did the job properly.

  Parameter: show

Step 4: Export chunks
  This step will use the conditioned chunk data and export the chunks. You can
  either export all chunks at once or select chunks. The tool will assembly all
  parts of the same recording into one file.
  Use paramater "show" to get the id of the chunk you want to extract. If you
  call export without any additional parameter, all chunks will be exported.

  Parameter: export


Additional Parameters:
----------------------

setup [setup-args]        Manages all settings necessary for a working script.

setup show                Show all settings.
setup reset               Reset all settings to default values.

setup input clear
setup input add [FILE]
setup input remove [FILE]

setup blocksize [INTEGER]
setup exportdir [STRING]
setup minchunksize [INTEGER]
setup maxcreategap [INTEGER]
setup maxsortgap [INTEGER]



Usage:
------

  usage
  setup [setup-args]
  create
  sort
  reset
  clear
  show
  export [chunk-id]


Tested devices:
---------------

  * Panasonic DMR-EH55
  * Panasonic DMR-EH56
  * Panasonic DMR-EH57
  * Panasonic DMR-EX77
  * Panasonic DMR-EX85
  * Panasonic DMR-XW300
  * Panasonic DVM-E80H

Download

Please note that there are multiple version of each download available. The *.tar.gz tarball contains the original and unmodified files. In the *.zip files the line endings are converted to the Windows style.

Stable releases: http://dl.dropbox.com/u/1996083/Projects/index.html (See directory “dvr-recover”.)

Git-Repository

git clone 'http://dl.dropbox.com/u/1996083/Projects/git/dvr-recover.git'

ChangeLog

This section may contain excerpts from files of the current development tree. The mentioned information might not match the behaviour of the current stable release completely.

2011-03-14  Stefan Haller  <haliner AT googlemail DOT com>

  Version 0.8:
  * dvr-recover.py: Don't drop the last block of n-1 chunk and first block of n
    chunk while splitting.
    Patch submitted by Alan Kan. Thanks.
  * dvr-recover.py, README: Added Panasonic DVM-E80H to list of working devices.
  * README: Some minor changes.


2011-01-26  Stefan Haller  <haliner AT googlemail DOT com>

  Version 0.7:
  * dvr-recover.py: Introduced fallback for determining file size.
    This solves issues with special files like device files.
    Thanks to Kevin Cardinal, who reported the bug and tested the developed fix.


2010-07-18  Stefan Haller  <haliner AT googlemail DOT com>

  Version 0.6.1:
  * dvr-recover.py: Fixed misplaced parenthesis introduced in release 0.6.
    Thanks to Martin Gašparík for pointing out this issue!


2010-07-11  Stefan Haller  <haliner AT googlemail DOT com>

  Version 0.6:
  * dvr-recover.py: Binary operations simplified.
  * dvr-recover.py: Fixed bug throwing exception "__main__.SqlManagerError:
    Multiple chunks are referencing the same chunk for concatenating!".
  * Readme: "Tuning" section added.


2010-06-17  Stefan Haller  <haliner AT googlemail DOT com>

  Version 0.5:
  * dvr-recover.py: Minor Bugfixes.
  * dvr-recover.py: Direct access to Windows device files forbidden.
  * dvr-recover.py: Use Sqlite3 database instead of chunk list file.
  * dvr-recover.py: All settings are stored in database now. Use the parameter
    "setup" to manipulate the preferences.


2010-05-11  Stefan Haller  <haliner AT googlemail DOT com>

  Version 0.4:
  * dvr-recover.py: test_settings: Don't print whole list of input filenames,
    print only one filename per line.
  * dvr-recover.py: create: Try to write empty chunk list before processing
    input file, so the script fails before the time-consuming process.


2010-05-10  Stefan Haller  <haliner AT googlemail DOT com>

  Version 0.3:
  * dvr-recover.py: Catch error while opening settings file.
  * dvr-recover.py: Don't load settings file for parameter sample_settings.
  * dvr-recover.py: Replaced term "option file" with "settings file" to
    reduce phrases with the same meaning.
  * dvr-recover.py: Added information concerning square brackets in usage
    message respectively docstring.


2010-05-09  Stefan Haller  <haliner AT googlemail DOT com>

  Version 0.2:
  * dvr-recover.py: FileReader class added.
  * dvr-recover.py: Handle multiple input streams.
  * dvr-recover.py: Docstrings added.


2010-04-29  Stefan Haller  <haliner AT googlemail DOT com>

  Version 0.1:
  * Initial Release

Current State of Development

This project is currently not under development. I succesfully recovered the hard drive disk of one of my relatives — the main work is done. At the moment mostly of the work are improvements — make the script better, faster and easier to use. As far as the project is useful for someone I try to maintain it and implement all requested features. If you have any additional suggestions or you’ve found any bugs, feel free to contact me!

Parts of this page have been generated automatically. Last update: Wed, 28 Sep 2011 11:00:06 +0200