Interaction with Shell

This tutorial summarizes useful tips for using Shell commands in Google Colab (which basically is IPython).

Prefixing with !

Any commands that work in the command-line can be used in Google Colab by prefixing it with !. For example, ls and bash can be run as depicted on the left hand side figure.

Beginning the cell with %%shell

The ! character requires us to prepend each line with !. If we want to interpret an entire cell as a shell script, we insert %%shell at the beginning of a cell (also, see the left hand side figure as an example).

Note: this Colab notebook shows some examples of them.

Passing Values to and from the Shell

We can interact the output of any Shell command with IPython. Following are examples:

content = !ls
print(content)

The above will print out ['sample_data'].

Note that this is handled by a special type defined in IPython called SList. To verify this, let's try:

type(content)

It will print out IPython.utils.text.SList.

Passing Variables into the Shell

Passing Python variables into the Shell can be done by the syntax {variable name}. For instance, 

message = "hello"
!echo {message}

The above prints out hello.

!cd does not work. Use the magic %cd

!cd cannot be used to navigate in Colab notebooks. The reason is that Shell commands are executed in a temporary sub-Shell. To handle this, you can use the magic %cd as follows: 

%cd /usr/local

The above navigates the file system to '/usr/local'.

%automagic for toggling magic functions

We can use %automatic to toggle Shell-like magic functions. After toggling, the Shell-like functions can be used without '%'. These functions supported by %automagic are %cd, %cat, %cp, %env, %ls, %man, %mkdir, %more, %mv, %pwd, %rm, and %rmdir.

Use %env to display a working environment

We can use %env to print out your working environment. Following is an example of executing it. 

{'CLICOLOR': '1', 
'CLOUDSDK_CONFIG': '/content/.config', 
'COLAB_GPU': '0', 
'CUDA_PKG_VERSION': '10-0=10.0.130-1', 
'CUDA_VERSION': '10.0.130', 
'CUDNN_VERSION': '7.6.0.64', 
'DATALAB_SETTINGS_OVERRIDES': '{"kernelManagerProxyPort":6000,"kernelManagerProxyHost":"172.28.0.3","jupyterArgs":["--ip=\\"172.28.0.2\\""]}', 
'DEBIAN_FRONTEND': 'noninteractive', 
'ENV': '/root/.bashrc', 
'GCS_READ_CACHE_BLOCK_SIZE_MB': '16', 
'GIT_PAGER': 'cat', 
'GLIBCPP_FORCE_NEW': '1', 
'GLIBCXX_FORCE_NEW': '1', 
'HOME': '/root', 
'HOSTNAME': '2b5001c68ac9', 
'JPY_PARENT_PID': '23', 
'LANG': 'en_US.UTF-8', 
'LAST_FORCED_REBUILD': '20190507', 
'LD_LIBRARY_PATH': '/usr/local/nvidia/lib:/usr/local/nvidia/lib64', 
'LD_PRELOAD': '/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4', 
'LIBRARY_PATH': '/usr/local/cuda/lib64/stubs', 
'MPLBACKEND': 'module://ipykernel.pylab.backend_inline', 
'NCCL_VERSION': '2.4.2', 
'NO_GCE_CHECK': 'True', 
'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 
'NVIDIA_REQUIRE_CUDA': 'cuda>=10.0 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=410,driver<411', 'NVIDIA_VISIBLE_DEVICES': 'all', 
'OLDPWD': '/', 
'PAGER': 'cat', 
'PATH':'/usr/local/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin:/opt/bin', 
'PWD': '/', 
'PYTHONPATH': '/env/python', 
'PYTHONWARNINGS': 'ignore:::pip._internal.cli.base_command', 
'SHELL': '/bin/bash', 
'SHLVL': '1', 
'TERM': 'xterm-color', 
'TF_FORCE_GPU_ALLOW_GROWTH': 'true', 
'_': '/tools/node/bin/node'}

How differences are % and %% as magics?

Both % and %% indicates magic commands but in different flavors. Character % indicates a line magic, which operates on a single line of input. On the other hand, character %% indicates a cell magic, which operates on multiple lines of input. The topmost figure exemplifies the use of %%writefile for writing outputs to a file.

Final Notes

Other blogged posts >> Tutorials