Abstract:

This presentation explores the feasibility and benefits of performing data manipulation and visualization directly within the terminal, advocating for its use due to "cool" factor, streamlined linear workflow, remote accessibility via SSH, and ease of integrating ASCII/Unicode plots into documentation. The speaker addresses challenges such as limited display scope for large datasets and plotting complexities.

The talk covers three main areas: tabular data, visualization, and interactive applications. For tabular data, tools like bat, xsv, and Visidata are demonstrated for enhanced viewing, sorting, and filtering of CSV and other formats. Python libraries like pandas, polars, and rich are shown for programmatic display of dataframes and aesthetically pleasing tables. For visualization, the terminal's capabilities (true color, Unicode) are highlighted, and simple DIY bar charts are presented. More advanced plotting is achieved using dedicated Python libraries such as plotille (using Braille characters for fine-grained lines) and plotext for line plots and histograms. A "cheat" method using the kitty protocol to embed matplotlib images directly in supporting terminals is also shown. Finally, for interactive applications and dashboards, the modern Python Textual library is introduced, along with the textual-fast-data-table and plotext widgets, demonstrating how to build interactive data browsers and simple dashboards within the terminal.

Summarizing Data Operations and Visualization in the Terminal

• 0:09 Introduction to Terminal Data Operations: The presentation focuses on the practicalities and benefits of working with and visualizing data directly within the command-line terminal.
• 0:56 Why Use the Terminal for Data? Key reasons include its "cool" factor, support for a linear workflow, utility in remote SSH environments where graphical interfaces are limited, and the ability to easily copy/paste ASCII-based plots into code documentation.
• 2:05 Challenges in Terminal Data Handling: The main difficulties include viewing large tables within a limited terminal window, limitations in plotting capabilities, and the need for more interactive user interfaces compared to traditional graphical tools.
• 3:18 Tabular Data Exploration:
  • 3:42 cat vs. bat: Standard cat is insufficient for CSVs; bat provides colored, visually separated columns for better readability.
  • 5:19 xsv and Visidata: xsv is introduced for advanced CSV operations like sorting and filtering via pipes. Visidata is presented as a powerful, full-featured data IDE that handles various formats (including Parquet) for interactive browsing, sorting, and filtering.
  • 8:31 Python Libraries for Tabular Data: pandas and polars dataframes print reasonably well by default, though large datasets may require output limits. The rich library can be used to generate visually appealing and colorful tables from dataframes.
• 11:47 Terminal Visualization/Plotting:
  • 12:07 Terminal Capabilities: Terminals support true colors, ASCII, Unicode symbols, and emojis, offering a surprisingly robust canvas for plotting.
  • 13:06 Alternate Screen: Terminals can operate in an "alternate screen" mode, allowing direct character placement without scrolling, which is crucial for interactive UI applications.
  • 13:30 Simple DIY Plots: Basic bar charts can be easily created with simple Python scripting for quick, readable output. Colors can be added using ANSI escape codes or libraries like rich.
  • 17:19 Dedicated Plotting Libraries: plotille (uses Braille characters for fine-grained lines) and plotext are highlighted as Python libraries capable of generating line plots and histograms directly in the terminal.
  • 20:51 matplotlib Integration (Cheat): A method is shown to embed matplotlib generated images directly into supporting terminals (e.g., Konsole on Linux) using the kitty protocol, allowing for more complex visualizations.
• 22:22 Interactive Terminal Applications (TUIs):
  • 22:51 Textual Library: A modern Python library for building rich terminal user interfaces (TUIs), supporting widgets, event binding, keyboard/mouse input, and CSS styling. The presentation itself is built with Textual.
  • 24:57 textual-fast-data-table: An external Textual widget crucial for displaying and browsing large datasets efficiently, as the default DataTable is limited.
  • 26:13 Embedding Plots in TUIs: plotext plots can be embedded into Textual applications using a dedicated plotext widget.
  • 26:41 Dashboard Example: A practical example of a Textual application is demonstrated, showing a weather data dashboard that combines a selectable table, interactive views, and dynamically updating plots.
• 27:39 Conclusion: The presenter concludes that it is entirely possible and relatively easy to manage tabular data, create various plots, and build simple interactive dashboard applications within the terminal using the discussed tools and libraries.