Interaction with Shell

Passing Values to and from the Shell

We can interact the output of any Shell command with IPython. Following are examples:

content = !ls

print(content)

The above will print out ['sample_data'].

Note that this is handled by a special type defined in IPython called SList. To verify this, let's try:

type(content) 

It will print out IPython.utils.text.SList.

Passing Variables into the Shell

Passing Python variables into the Shell can be done by the syntax {variable name}. For instance,

message = "hello"

!echo {message}

The above prints out hello.

!cd does not work. Use the magic %cd

!cd cannot be used to navigate in Colab notebooks. The reason is that Shell commands are executed in a temporary sub-Shell. To handle this, you can use the magic %cd as follows:

%cd /usr/local

The above navigates the file system to '/usr/local'.

%automagic for toggling magic functions

We can use %automatic to toggle Shell-like magic functions. After toggling, the Shell-like functions can be used without '%'. These functions supported by %automagic are %cd, %cat, %cp, %env, %ls, %man, %mkdir, %more, %mv, %pwd, %rm, and %rmdir.

Use %env to display a working environment

We can use %env to print out your working environment. Following is an example of executing it.

{'CLICOLOR': '1', 

'CLOUDSDK_CONFIG': '/content/.config', 

'COLAB_GPU': '0', 

'CUDA_PKG_VERSION': '10-0=10.0.130-1', 

'CUDA_VERSION': '10.0.130', 

'CUDNN_VERSION': '7.6.0.64', 

'DATALAB_SETTINGS_OVERRIDES': '{"kernelManagerProxyPort":6000,"kernelManagerProxyHost":"172.28.0.3","jupyterArgs":["--ip=\\"172.28.0.2\\""]}', 

'DEBIAN_FRONTEND': 'noninteractive', 

'ENV': '/root/.bashrc', 

'GCS_READ_CACHE_BLOCK_SIZE_MB': '16', 

'GIT_PAGER': 'cat', 

'GLIBCPP_FORCE_NEW': '1', 

'GLIBCXX_FORCE_NEW': '1', 

'HOME': '/root', 

'HOSTNAME': '2b5001c68ac9', 

'JPY_PARENT_PID': '23', 

'LANG': 'en_US.UTF-8', 

'LAST_FORCED_REBUILD': '20190507', 

'LD_LIBRARY_PATH': '/usr/local/nvidia/lib:/usr/local/nvidia/lib64', 

'LD_PRELOAD': '/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4', 

'LIBRARY_PATH': '/usr/local/cuda/lib64/stubs', 

'MPLBACKEND': 'module://ipykernel.pylab.backend_inline', 

'NCCL_VERSION': '2.4.2', 

'NO_GCE_CHECK': 'True', 

'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 

'NVIDIA_REQUIRE_CUDA': 'cuda>=10.0 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=410,driver<411', 'NVIDIA_VISIBLE_DEVICES': 'all', 

'OLDPWD': '/', 

'PAGER': 'cat', 

'PATH':'/usr/local/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin:/opt/bin', 

'PWD': '/', 

'PYTHONPATH': '/env/python', 

'PYTHONWARNINGS': 'ignore:::pip._internal.cli.base_command', 

'SHELL': '/bin/bash', 

'SHLVL': '1', 

'TERM': 'xterm-color', 

'TF_FORCE_GPU_ALLOW_GROWTH': 'true', 

'_': '/tools/node/bin/node'}

How differences are % and %% as magics?

Both % and %% indicates magic commands but in different flavors. Character % indicates a line magic, which operates on a single line of input. On the other hand, character %% indicates a cell magic, which operates on multiple lines of input. The topmost figure exemplifies the use of %%writefile for writing outputs to a file.

Final Notes

• You can see examples of using Shell commands in Google Colab in this Colab notebook.
• You can see some of IPython Magic commands in this Python Data Science handbook.