### dvr-recover

This section may contain excerpts from files of the current development tree. The mentioned information might not match the behaviour of the current stable release completely.

      ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::      _                                                         ::
::   __| |_   __ _ __      _ __  ___   ___  _____   __ ___  _ __  ::
::  / _ \ \ / /| '__|____| '__|/ _ \ / __|/ _ \ \ / // _ \| '__| ::
:: | (_| |\ V / | | |_____| |  |  __/| (__| (_) \ V /|  __/| |    ::
::  \__,_| \_/  |_|       |_|   \___| \___|\___/ \_/  \___||_|    ::
::                                                                ::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Description
~~~~~~~~~~~~~

dvr-recover is an incredible Python-powered project helping you to recover
recordings of digital video recorders. In other words: The script is able to
find video files on the surface of the device's hard disk drive.

Practically, the process is much more complex. Have a look at the following
sections if you want to gain more knowledge about the details.

There are many devices on the market, not all models can be supported. See the
list of tested devices to see if your device has been already tested.

~~~~~~~~~

This program is free software: you can redistribute it and/or modify
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

Tested Devices
~~~~~~~~~~~~~~~~

Currently only Panasonic DMR devices have been tested. If you successfully
recovered the content of your drive (completely or partly), please drop me a
mail!

Working devices:

* Panasonic DMR-EH55
* Panasonic DMR-EH56
* Panasonic DMR-EH57
* Panasonic DMR-EX77
* Panasonic DMR-EX85
* Panasonic DMR-XW300
* Panasonic DVM-E80H

System Requirements
~~~~~~~~~~~~~~~~~~~~~

To run the script you need a Python interpreter (2.x, tested with 2.5 and
higher).

Usage
~~~~~~~

If you execute the script without any parameters, the script will print the
usage message where all parameters, switches and settings are explained in
detail.

Executing the script is very easy: Simply open a terminal (on Windows:
[Win] + [R] -> enter "cmd" and press enter). At first you have to switch to the
directory where the script is located and then you can fire up the Python
interpreter.

Linux:

cd /path/to/script/
python dvr-recover.py [parameters]

Windows:

cd C:\Path\to\script\
python dvr-recover.py [parameters]

Guide
~~~~~~~~

This section will guide you through the complete process of recovering the
data from you hard disk drive.

Naturally the hard disk drive of the digital video recorder must be plugged
into the computer. So you need to open the device and fetch the hdd. You can
either attach the disk directly to the IDE ports of your motherboard or you
can use a IDE-USB-bridge.

It's recommended that you create an image of your disk, but if you think you're
lucky you can even use the hard disk drive directly as input on Linux. Note
that direct access to the hard disk drive on Windows is currently not supported
by dvr-recover. On Windows you have to use 3rd party programs to create the
image. On Linux you can use the standard tools (be sure that you have root
permissions):

(Assuming that your disc is labeled /dev/sdb)
dd if=/dev/sdb of=disk.img bs=10M

This guide assumes that you have an exact bitwise copy of the hdd on your
file system. Note that it's also possible to split your image into smaller
parts. The script is able to handle this situation correctly. But you have to
be sure that you specify all parts in the correct order in the later process.
If you want to use the hdd directly, you will need to substitute the filenames
in this guide with the device identifier (/dev/sdb, ... on Linux).

The script will automatically create a sqlite3 database to store all
settings and gained information. If somethings goes heavily wrong, you will
be able to reset the state of the program by deleting the database. To do
so delete the file "dvr-recover.sqlite" in the working directory of the
script.

Before you the script can analyze your hard disk drive, you have to go through
a initial setup first. The script needs at least the information about the
input filename(s). But there are many other settings to tweak -- most of them
the script by passing "usage" as parameter.

If you call the script with passing "setup" as parameter, it will print the
current values of all settings:

python dvr-recover.py setup
|  input_file: /home/user/disk.img.0001
|  [...]
|  input_file: /home/user/disk.img.9999
|  export_dir: /tmp/export
|  blocksize: 2048
|  min_chunk_size: 25600
|  max_create_gap: 90000
|  max_sort_gap: 90000

To change a setting you have to pass the "setup" parameter and additionally the
setting to change with the new value:

python dvr-recover.py setup blocksize 2048

The only exception is the handling of input files:

python dvr-recover.py setup input clear
python dvr-recover.py setup input add disk.img.0001
python dvr-recover.py setup input add disk.img.0002
python dvr-recover.py setup input del disk.img.0001
[and so on...]

Use the parameter "clear" to clear the list of input files. Also available
are the parameters "add" and "del" -- both take a filename as argument.

If you want to reset all settings, you have to use the parameter "reset":

python dvr-recover.py setup reset

Now you should be able to start the main feature of the script: recovering
you hard disk drive.

At first let the script analyze your hard disk drive. Call the script
with passing the parameter "create". This process will take quite a long
time to complete. You can interrupt the process at any point by pressing
[CTRL] + [C]. On the next call with the parameter "create" the script will
automatically resume the process where it was interrupted.

python dvr-recover.py create
|  [ 29.5%] 297457/1006929 blocks (9915.2 bl/s; 19.4 MiB/s): 5 chunks
|  [ 65.0%] 654794/1006929 blocks (11911.2 bl/s; 23.3 MiB/s): 6 chunks
|  [ 91.3%] 918841/1006929 blocks (8786.1 bl/s; 17.2 MiB/s): 6 chunks
|
|  Finished.
|  Read 1006929 of 1006929 blocks.
|  Found 7 chunks.
|  Took 100.97 seconds.
|  Average speed was 9972.6 blocks/s (19.5 MiB/s).

(The file used for generating the output above was very very small.)

After finishing the analyze of the hdd the script will forbid all further
calls with the parameter "create", because this would cause data loss
(all data of the last analyze would be reseted first). You must explicitly
reset the information in the database by passing the parameter "clear":

python dvr-recover.py clear

(THIS WILL DELETE ALL GATHERED DATA FROM THE DATABASE!)

dvr-recover has the ability to sort the chunks and find chunks of the same
recording. To start this process pass the paremter "sort" to the script:

python dvr-recover.py sort

To revert the effect of "sort", pass "reset".

python dvr-recover.py reset

At every stage you can display the chunk info by passing "show".

python dvr-recover.py show
|  ----+--------------+--------------+--------------+--------------+------------
|      |  Block Start |   Block Size |  Clock Start |    Clock End | Concatenate
|  ----+--------------+--------------+--------------+--------------+------------
|    1 |       391379 |       615548 |    250031969 |    402836926 |     False
|    2 |       223188 |       168189 |    401478251 |    484558534 |     False
|    3 |       107989 |       115197 |    546892241 |    592303341 |     False
|    4 |        80992 |        26995 |   1085209725 |   1100156779 |     False
|    # |        53995 |        26995 |   1100157071 |   1115282776 |      True
|    # |        26998 |        26995 |   1115283069 |   1129877061 |      True
|    # |            0 |        26996 |   1129877208 |   1144930337 |      True

In the first column you can see the index of the chunk. A '#' is printed if the
chunk is concatenated to the previous one.

The last thing is exporting the recordings. If the scrit has found chunks
of the same recording, they will be automatically concatenated. You can
either export all recordings in the database or only a single one.

python dvr-recover.py export

This will export all recordings. You can also specify a chunk id to export. In
this example the script would export only the chunk with the id 4 (including
the chunks which should be concatenated to this one).

python dvr-recover.py export 4

All the basics are explained -- you should be able to use the script now. Of
course, there are a lot of other things that were not mentioned in this guide.
For more details have a look at the usage message or the source code.

Tuning the parameters
~~~~~~~~~~~~~~~~~~~~~~~

Is there any possibility to tune the script and recover more files? Yes, it is!

You can tune the process of chunk detection with parameters. The most important
parameter is "min_chunk_size". The smaller the value is, the more the script
finds. But the problem is that the possibility to find "garbage" is even
higher.

The default setting is that the program ignores chunks smaller than 50 MB
(25600 blocks). If you set it to 0, the script will find everything, even if
it's only one frame of a recording. If you play a bit with this parameter you
maybe will recover more recordings completely.

IMPORTANT: After changing the parameters run the complete scan again (parameter
"create")!

To change the setting run:

python dvr-recover.py setup min_chunk_size [NEW_VALUE]

[NEW_VALUE] must be given in blocks.
Blocks * blocks size = size in byte. The default block size is 2048 byte.

I tried to set the default value to a sensible default. If somebody can prove
that a smaller value is better, I'll set the default value for later releases
to a lower value.

Homepage and Contact
~~~~~~~~~~~~~~~~~~~~~~

Stefan Haller <haliner AT googlemail DOT com>

Feel free to contact me if you have any additional suggestions or you've
found a bug. Feedback is always welcome! IF YOU THINK THE SCRIPT IS TERRIFIC,
I WOULD BE VERY GLAD IF YOU LET ME KNOW THAT.


## Installation

This section may contain excerpts from files of the current development tree. The mentioned information might not match the behaviour of the current stable release completely.

There is no need to install dvr-recover. Simply extract the tarball or the
zipfile anywhere and execute the script.



## Usage

This section may contain excerpts from files of the current development tree. The mentioned information might not match the behaviour of the current stable release completely.

dvr-recover - extract MPEG2 files of digital video recorder hdd
===============================================================

Version: 0.6

This program is free software: you can redistribute it and/or modify
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

Sqlite database file
--------------------

The script will create a sqlite3 database in the current directory with the
name "dvr-recover.sqlite". All settings and data will be stored in this
database.

Setup
-----

Before you can use dvr-recover you will have to change the default settings.
You can do this by simply calling the script with the parameter "setup"
followed by the setting to change and the new value. To change the value of
the blocksize setting call:

python dvr-recover.py setup blocksize 2048

Here is a full listing of all available settings:

The square brackets in this listing are representing the type of the value.
Replace them with an integer for "[integer]". Don't copy this square
brackets, so write

blocksize 2048

blocksize [2048]

input clear            This option sets the path of the hard disk drive
input add [filename]   file used as input. You can either use a copy of the
input del [filename]   block device (created with something like dd or so) or
the block device directly (required root privileges).
The file must be readable. It's possible to specify
multiple hdd-files by calling the add parameter
multiply times. The script will threat the single files
as one big file. That way you can split the hdd into
smaller pieces.

parameter clear:   clear list of input file
parameter del:     delete one file from list of input
files

export_dir [string]    Defines where the output should be written to. Must
match an existing path. Both relative and absolute
paths are accepted. Current directory is "./".

blocksize [integer]    The blocksize of the filesystem can be configured
with this option. Set this to a value in bytes.
The default value is 2048 bytes. Probably this value
should work, but if not, you're free to tune it.

min_chunk_size [integer]
If the script finds chunks smaller than this size
(value must be given in blocks!), it will ignore them
silently. If this value is too small, the script
will find chunks that were deleted or can't be used.
Otherwise, if the value is too big, valuable chunks
will be ignored. The default value is 25600 blocks
(50 MiB by blocksize of 2048 bytes).

max_create_gap [integer]
The script will split the stream into two chunks if
it finds two frames where the timecode differs more
than this value. MPEG uses a clock of 90 kHz.
So the default value of 90,000 ticks equals one second.

max_sort_gap [integer]
See maxcreategap. This value is used to concatenate
two chunks if the difference of the timecode is smaller
than this value. The default value of 90,000 ticks
equals one second.

Input hdd file
--------------

You can create either a copy of the hdd or use the hdd directly as input.

Linux:
If you want to copy the hdd (assuming the hdd is /dev/sdb) then you can use
this dd command: (setting the blocksize to 10MB should increase performance.)

dd if=/dev/sdb if=hddfile bs=10MB

If you want to use the hdd directly as input, use this as value for hdd-file:
(Assuming that the hdd is /dev/sdb.)

hdd-file=/dev/sdb

Windows:
Use a tool to create a copy of the hdd. Direct access to Windows device
files (\\.\PhysicalDrive0, ...) is not supported at the moment.

Steps:
------

Step 1: Export information of chunks
The hdd file will be analyzed and all necessary information are collected.
This step may take quite a long time, be patient! This step is only necessary
once. All other steps will use the stored chunk info to save time.
(The script tries to find mpeg headers and extract the timecode. Depending on
the timecode it's possible to split the stream into separate chunks.)

Parameter: create

Step 2: Analyze and sort chunks
This step will analyze the stored chunk info and sort the chunks. The tools
tries to find parts of the same recording (by analyzing the timecode
informationof the chunks) and bring them into the right order.

Parameter: sort

Step 3: Show chunks
You can list all chunks to make sure that the program did the job properly.

Parameter: show

Step 4: Export chunks
This step will use the conditioned chunk data and export the chunks. You can
either export all chunks at once or select chunks. The tool will assembly all
parts of the same recording into one file.
Use paramater "show" to get the id of the chunk you want to extract. If you
call export without any additional parameter, all chunks will be exported.

Parameter: export

----------------------

setup [setup-args]        Manages all settings necessary for a working script.

setup show                Show all settings.
setup reset               Reset all settings to default values.

setup input clear
setup input remove [FILE]

setup blocksize [INTEGER]
setup exportdir [STRING]
setup minchunksize [INTEGER]
setup maxcreategap [INTEGER]
setup maxsortgap [INTEGER]

Usage:
------

usage
setup [setup-args]
create
sort
reset
clear
show
export [chunk-id]

Tested devices:
---------------

* Panasonic DMR-EH55
* Panasonic DMR-EH56
* Panasonic DMR-EH57
* Panasonic DMR-EX77
* Panasonic DMR-EX85
* Panasonic DMR-XW300
* Panasonic DVM-E80H



Please note that there are multiple version of each download available. The *.tar.gz tarball contains the original and unmodified files. In the *.zip files the line endings are converted to the Windows style.

Stable releases: http://dl.dropbox.com/u/1996083/Projects/index.html (See directory “dvr-recover”.)

## Git-Repository

git clone 'http://dl.dropbox.com/u/1996083/Projects/git/dvr-recover.git'

## ChangeLog

This section may contain excerpts from files of the current development tree. The mentioned information might not match the behaviour of the current stable release completely.

2011-03-14  Stefan Haller  <haliner AT googlemail DOT com>

Version 0.8:
* dvr-recover.py: Don't drop the last block of n-1 chunk and first block of n
chunk while splitting.
Patch submitted by Alan Kan. Thanks.

2011-01-26  Stefan Haller  <haliner AT googlemail DOT com>

Version 0.7:
* dvr-recover.py: Introduced fallback for determining file size.
This solves issues with special files like device files.
Thanks to Kevin Cardinal, who reported the bug and tested the developed fix.

2010-07-18  Stefan Haller  <haliner AT googlemail DOT com>

Version 0.6.1:
* dvr-recover.py: Fixed misplaced parenthesis introduced in release 0.6.
Thanks to Martin Gašparík for pointing out this issue!

2010-07-11  Stefan Haller  <haliner AT googlemail DOT com>

Version 0.6:
* dvr-recover.py: Binary operations simplified.
* dvr-recover.py: Fixed bug throwing exception "__main__.SqlManagerError:
Multiple chunks are referencing the same chunk for concatenating!".

2010-06-17  Stefan Haller  <haliner AT googlemail DOT com>

Version 0.5:
* dvr-recover.py: Minor Bugfixes.
* dvr-recover.py: Use Sqlite3 database instead of chunk list file.
* dvr-recover.py: All settings are stored in database now. Use the parameter
"setup" to manipulate the preferences.

2010-05-11  Stefan Haller  <haliner AT googlemail DOT com>

Version 0.4:
* dvr-recover.py: test_settings: Don't print whole list of input filenames,
print only one filename per line.
* dvr-recover.py: create: Try to write empty chunk list before processing
input file, so the script fails before the time-consuming process.

2010-05-10  Stefan Haller  <haliner AT googlemail DOT com>

Version 0.3:
* dvr-recover.py: Catch error while opening settings file.
* dvr-recover.py: Don't load settings file for parameter sample_settings.
* dvr-recover.py: Replaced term "option file" with "settings file" to
reduce phrases with the same meaning.
* dvr-recover.py: Added information concerning square brackets in usage
message respectively docstring.

2010-05-09  Stefan Haller  <haliner AT googlemail DOT com>

Version 0.2:
* dvr-recover.py: Handle multiple input streams.
`