apb Manual - A Pileup Browser

v0.5.0

This is the manual for apb version 0.5.0. This manual is generated directly from the apb binary. If you need a different version of the manual, please check the helptext for instructions as to how to generate the manual for your version of the tool.

If you are reading this manual from a file on disk (rather than the webpage hosted with the apb GitHub repo), it is recommended to read this manual using a markdown viewer. If you are reading on a cluster and do not have a markdown viewer, try a pager such as less. If you have a sufficiently recent version of less, you can use less --wordwrap for an improved experience.

apb is subject to change pending user feedback until the 1.0 release. Some aspects of the tool are unpolished and quirky - I will continue to work on those. In the meantime please send any feedback via GitHub issues at https://github.com/blex-max/apb-pileup-browser, where you will also find the README for installation instructions, the project roadmap, and other background.

1) Overview

Given one or more alignment files and a genomic locus apb retrieves all reads aligned to that locus and loads the reads into a fast, queryable database. The TUI then displays the aligned reads alongside user-selected data for each read (e.g. mapping quality, leftmost alignment position, etc.), and provides a command prompt at which you can enter commands to query the reads or change the display. The display and command prompt can be navigated via simple arrow key navigation.

Important

apb displays all coordinate data in 1-based closed coordinates. This includes the LOCUS argument passed at startup to the CLI.

2) CLI Usage

See the helptext from apb --help for the canonical guide to the CLI. Some additional context is given here.

apb LOCUS FILE... opens one or more alignment files (SAM/BAM/CRAM) at a locus (chr1:12345), loads the pileup, and launches the TUI. A locus is specified in the form contig:coordinate - only a single coordinate needs to be provided, rather than a length-1 range as in many samtools commands. If a reference is provided via --ref PATH, the reference will be shown and the alignment pane will be enriched to indicate divergence from reference in the displayed reads.

apb --demo runs against synthetic data generated by apb and does not require any file input. It is useful for learning apb, but note that since the data is artifically generated it is somewhat unrealistic.

apb --db DB reopens a database file previously produced by --dump (or the in-TUI dump command), and does not take any other file input.

3) TUI Usage

On starting apb you will enter the TUI. The TUI is split into 2 major regions. At the top you have the read viewer, made up of an alignment pane on the left and a table pane of per-read data on the right. Information about the pileup locus is displayed at the top left of the read viewer. At the bottom you have a command prompt at which you can enter commands to query the reads and control the display. Commands are typed at the command prompt >, and return messages are displayed below at the double chevron. Queries will be displayed at the top right of the prompt, including the count of reads matching the query. In the alignment pane, = matches the reference, x is a deletion, and a ring over a base marks an insertion, displayed beneath at ^.

3.1) Basic Navigation

apb has a modal interface, which means that the way the program responds to your input depends on which mode you are in. There are only two major modes: viewer, for viewing the reads in the read viewer, and prompt, for controlling the command prompt. You can switch between modes with the Esc key, and the current mode is always beneath the read viewer as focused (Esc): <mode>.

In viewer mode, the read viewer is scrolled, and the alignment pane panned, with the arrow keys. Mouse scroll may also work if your terminal is configured to send mouse scrolling as arrow keys, which is reasonably common. In prompt mode, normal typing goes directly to the prompt at the >, and Enter dispatches a typed command. Keyboard shortcuts are provided for easily navigating and editing text while in prompt mode.

The following reference table is also available via help nav at the prompt:

 M-: Alt | C-: Ctrl                                 
                                                    
 ALWAYS ACTIVE:                                     
  Esc               change focus/mode between the   
                    read viewer and the prompt      
  C-c               quit apb                        
                                                    
 FOCUS: VIEWER                                      
  Up / Down         scroll one row                  
  Left / Right      pan alignment pane side-to-side 
  PgUp / PgDn       scroll one page                 
                                                    
 FOCUS: PROMPT                                      
  C-w               recall where clause to command  
                    line                            
  C-o               recall order by clause to       
                    command line                    
  C-/               undo last modification to       
                    query                           
                                                    
  Enter             submit command                  
  Left / Right      move cursor                     
  Up / Down         scroll command history          
  C-Left / C-Right                                  
  M-Left / M-Right                                  
  M-b / M-f         back / forward one word         
  C-a / C-e                                         
  Home / End        start / end of line             
  Bksp / C-d        delete char back / forward      
  M-Bksp / M-d      delete word back / forward      
  C-k / C-u         delete to end / start of line   

3.2) Command Reference

Every command submitted at the prompt is interpreted like <command> [args]. Shorthand aliases are provided for many commands for quick use.

The following reference table is also available via help cmd at the prompt.

 COMMAND REFERENCE
  `help [(nav|cmd|col)]`:
    Show help for given topic,
    or general help with no args.
     aliases: h, ?

  `quit `:
    Exit apb.
     aliases: q, exit, close

  `where [condition]`:
    Start a new WHERE clause, overwriting
    any existing condition. Omitting the
    [condition] argument will clear the current
    WHERE clause. For a reference of queryable
    columns, check the manual or run `? col`.
     aliases: w, wh, WHERE

  `order by [condition]`:
    Sort reads by ORDER BY expression. Clears order
    by with no args, resetting to default ordering.
    When viewing multiple alignments, reads are
    always sorted by file as the primary key.
     aliases: o, order, ORDER BY, sort

  `count <condition>`:
    Count reads matching <condition>, AND-concatenated
    onto the current WHERE clause if present.
     aliases: ct

  `count all <condition>`:
    Count all reads matching <condition>
     aliases: cta

  `clear `:
    Clear current query, resetting
    both WHERE and ORDER BY.
     aliases: cl

  `col <field-name>...`:
    Show/hide read data columns in the
    table pane. For a list of available
    columns, check the manual or run `? col`.
     aliases: c

  `toggle (quality|insertion|table|sparkline)`:
    Toggle various aspects of the display.
    - quality:   read quality track in
                 alignment pane.
    - insertion: read insertion track in
                 alignment pane.
    - table:     table pane.
    - sparkline: switch quality track
                 between Phred ASCII and a
                 quick-look sparkline graph.
    `toggle` accepts any unambiguous substring
    of the argument, e.g. `tg qual`.
     aliases: tg

  `dump <path>`:
    Write the pileup database to an sqlite3 dump file.

3.3) Column Reference

For each read, the database stores the columns detailed below. All columns are queryable in where/order by/count commands. If the content of a column is not clearly displayed by the alignment pane, the column can be displayed in the table pane.

The following reference table is also available via help col at the prompt.


 TABLE REFERENCE                                
                                                
 The following columns are available for        
 querying. `qname` through `mstart` may be      
 displayed in the table pane with the `col`     
 command.                                       
                                                
  `path`:                                       
      text                                      
      Path to alignment file.                   
  `seq`:                                        
      text / NULL                               
      the read sequence. NULL if missing.       
  `qual`:                                       
      text / NULL                               
      the read Phred base qualities. NULL if    
      missing.                                  
  `base`:                                       
      character / NULL                          
      Nucleotide at the pileup position. NULL   
      if base is deleted, or `seq` is missing.  
  `indel`:                                      
      integer                                   
      indel length to the next mapped           
      nucleotide in the read.                   
      (0 = no indel, >0 insertion, <0 deletion) 
  `qname`:                                      
      text / NULL                               
      the read/template name.                   
      NULL if missing.                          
  `flag`:                                       
      integer (bitfield)                        
      SAM bitwise FLAG.                         
  `start`:                                      
      integer > 0                               
      1-based leftmost mapping position of      
      read.                                     
  `end`:                                        
      integer >= start                          
      1-based rightmost mapping position of     
      read.                                     
  `mapq`:                                       
      integer 0-255                             
      read mapping quality.                     
  `basequal`:                                   
      integer / NULL                            
      Phred base quality at the pileup          
      position. NULL if base is deleted, or     
      `qual` is missing.                        
  `qpos`:                                       
      integer > 0                               
      1-based offset into `seq`/`qual` to       
      pileup locus position. In deletions, this 
      is the first base after the deletion.     
  `cigar`:                                      
      text                                      
      CIGAR string of alignment operations.     
  `mtid`:                                       
      text / NULL                               
      reference name of the mate/next read.     
      = if mate is on same contig. Usually = if 
      mate is unmapped. Usually NULL if read    
      unpaired.                                 
      (use `flag` to check if mate is unmapped) 
  `mstart`:                                     
      integer > 0 / NULL                        
      mate/next read's leftmost mapping         
      position. Usually identical to `start`    
      if mate is unmapped. Usually NULL if read 
      unpaired.                                 
      (use `flag` to check if mate is unmapped) 
  `tags`:                                       
      text (JSON) / NULL                        
      aux tags as JSON; able to be individually 
      queried using ->> operator. NULL if tags  
      are missing.                              
  `is_del`:                                     
      integer 1-0 (true/false)                  
      1 if `base` is a deletion.                
  `is_head`:                                    
      integer 1-0 (true/false)                  
      1 if `qpos` is the read's first aligned   
      base.                                     
  `is_tail`:                                    
      integer 1-0 (true/false)                  
      1 if `qpos` is the read's last aligned    
      base.                                     
  `is_refskip`:                                 
      integer 1-0 (true/false)                  
      1 if `qpos` is a reference skip.          

For a guide to the symbols used in the quality string, see https://en.wikipedia.org/wiki/Phred_quality_score#Symbols. Note that toggle sparkline switches the quality track in the alignment pane between ASCII display of quality scores, and a sparkline graph - i.e. FAFF-FF<FFI vs ▆▅▆▆▁▆▆▄▆▆▆.

indel may require further explanation. Essentially, if the base at the pileup position is followed by an indel, then indel will contain the size of that indel event. A deletion is represented by a negative size (bases lost), and an insertion is represented by a positive size (bases gained).

For advanced users, note that most of these map directly onto fields in htslib’s bam_pileup1_t and bam1_t structs - consult the schema.

3.4) Querying the Pileup

The apb query commands provide a simple wrapper around (sqlite3-based) SQL. This means you can directly query the pileup to view and count reads by their properties, such as the base, flag, and cigar string. apb provides three major commands for querying - where, for filtering the pileup to reads matching a condition; order by, for sorting or grouping reads in the view; and count, for quantifying reads. This is best explained by example:

If this is your first time using apb, you may find it useful to walk through the following example in the demo mode, which can be accessed at the CLI by apb --demo. The numbers will not exactly match the examples as the examples are only illustrative.

Imagine we are inspecting a candidate SNV variant locus. We may start by filtering the pileup to only reads which the caller should consider. Specifics will vary by caller, but a common starting point is “reads which map well, have mapped mates, and aren’t duplicates, qc fails, or secondary/supplementary alignments”. At the command prompt we enter:

where mapq > 0 AND (flag & 3848) = 0;

This is an SQL WHERE clause - essentially a filter. The read viewer will update to show only reads matching that filter, and the status line will now show:

- WHERE mapq > 0 AND (flag & 3848) = 0: 76/100 reads

Let’s say our candidate SNV is A->T. We can sort the reads to group reads expressing the variant base at the pileup locus and view them in the read viewer:

order by base IS 'T' DESC

This is an SQL ORDER BY clause - essentially a sort. DESC is simply the SQL keyword for descending order (ASC is for ascending, which is the default). DESC puts reads that match our condition to the top of the sort for each alignment file in the pileup. The status line will now show:

- WHERE mapq > 0 AND (flag & 3848) = 0: 76/100 reads │ ORDER BY base IS 'T' DESC

The read viewer now shows only reads matching our WHERE condition, grouped by our ORDER BY condition. We may now explore the pileup visually for any properties of interest, e.g. if most of the reads supporting the variant are soft-clipped. Switch to the read viewer with Esc, and switch back to the prompt with Esc again when finished.

After exploring the pileup, it can be useful to quantify the locus in more detail. For example, to count the number of reads exhibiting the variant:

count base IS 'T'

This will return a message below the prompt at »:

- base IS 'T' / mapq > 0 AND (flag & 3848) = 0: 24/76 reads

The count returned is the count of reads matching the current WHERE condition AND the condition provided to count, against the total count of reads matching the current WHERE condition. If instead you wanted to count all reads matching a condition and ignoring the current WHERE condition, use count all [condition]:

count all base IS 'T'

returning:

- base IS 'T': 28/100 reads

From this we may conclude that four reads exhibit the variant, but are filtered out by our current WHERE clause on mapq and flag. We may consider relaxing our requirement that the mate must be mapped to see if this brings any of those four reads into view. To do so, we can simply grab the current WHERE clause from the status bar with Ctrl-w. This recalls it to the prompt, where we may edit and resubmit:

where mapq > 0 AND (flag & 3840 /* edited from 3848 */ ) = 0;

Editing the query is made easy by using the editing key shortcuts detailed in Section 3.1. This concludes the example, but is by no means the limit of what you can do with apb. See the next section for further examples and snippets.

Important

flag is a bitmask of read properties defined by the SAM format - see https://broadinstitute.github.io/picard/explain-flags.html from the Broad Institute, which both explains the meaning of the flag bits and allows you to calcuate the number (like 3848) which corresponds to a particular set of properties. See https://www.w3schools.com/programming/prog_operators_bitwise.php for a quick introduction to bitmask syntax. SQLite supports the bitwise and, or, and not.

Important

In SQL the ORDER BY clause can take multiple keys. This is supported by the order by command, so we could for example have written order by base IS 'T' DESC, basequal. This would have sorted the view such that reads exhibiting a ‘T’ at the pileup locus will appear first, but now within that group, reads will be sorted by the quality of that base, in ascending order (lowest first).

Important

Missing values are encoded as NULL, and different operators behave differently with NULL values in SQL. For example, a deleted base is NULL - where base != 'A' will exclude deletions, as NULL always evaluates to false in comparisons using !=, i.e. NULL != 'A' is false. Whereas, where base IS NOT 'A' will include deletions as NULL IS NOT 'A' evaluates to true. To check for the presence of missing values for a field, use where <col> IS NULL.

Important

In SQL, AND conditions always bind more tightly than OR. This means that:

WHERE a AND b AND c OR d

is equivalent to:

WHERE (a AND b AND c) OR d

If you want to bind two conditions together, use brackets:

WHERE a AND b AND (c OR d)

3.5) Further Querying Examples

Querying is flexible and powerful. Some useful examples:

Querying by Path

When viewing multiple alignment pileups, use instr() to subset your query to a specific file, e.g.:

where instr(path, 'PD12345a') and base != 'A'

Indels at the Pileup Locus

This is made quite simple by the indel column. For example, to search for a 3-base insertion anchored at the pileup:

where indel = 3

deletions are represented by negative numbers, so for a 3-base deletion:

where indel = -3

Note that these commands search for indels anchored at the pileup base, i.e. beginning immediately after the pileup locus. If you want to search for deleted bases at the pileup locus itself, use:

where is_del

Aux Tags

Aux tags may be queried with ->>. For example, to subset a query to a read group via the RG tag:

where base != 'G' and tags ->> '$.RG' = 'sample1'

For numeric tags, all comparison an arithmetic operators are available:

where tags ->> '$.NM' > 2

A read missing a tag returns null, so:

where tags ->> '$.RG' is null

Would find reads missing the RG tag.

CIGAR Querying

Pattern matching is available on all text fields, such as the CIGAR string. E.g. to filter for reads containing a 3-base insertion at any position:

where instr(cigar, '3I')

or for reads containing soft clipping of any length:

where instr(cigar, 'S')

Read Length and Arithmetic

Arithmetic functions are available. For example:

where end - start < 100

would filter the view to reads less than 100 bases long.

Motifs at the Query Position

This is a slightly more advanced example. qpos is the 1-based offset into seq for the base at the pileup position, matching SQLite’s own 1-based substr() indexing, so to search for the 4-mer GATC starting at the query position:

where substr(seq, qpos, 4) = 'GATC'

LIKE (_/%) and GLOB (?/*/[ACG]) both work as wildcards. To search for two possible trinucleotide motifs at the query position:

where substr(seq, qpos, 3) glob 'A[CG]T'

You can also search for motifs within a window of the seq string. This command searches for GATC within the first 10 bases of the read:

where instr(substr(seq, 1, 10), 'GATC') > 0

Any and all of these approaches may be combined, and more is possible. See SQLite’s expression/function reference - https://sqlite.org/lang_expr.html.


Addenda & Troubleshooting

The dump Functionality

A dump is a small, self-contained sqlite3 file with just the reads at the specified locus. A few use cases:

Note that all coordinate data within the dump is 1-based, matching the display.

A Word on Indexing Systems

htslib/samtools/bcftools, and by extension all alignment and VCF data, mix 0-based half-open and 1-based closed coordinate systems. This can be tricky to navigate.

apb uses 1-based closed coordinates throughtout, i.e coordinates start at 1, and coordinate ranges are [x-y] where both x and y are included. 1-based closed coordinates match the human-readable VCF and SAM format specs, and to samtools commands e.g. samtools view .... Coordinates from these sources, i.e. a variant locus from a VCF, may be passed directly to the locus CLI argument at startup. Loci from 0-indexed sources, e.g. BED files, may be passed directly to the CLI argument if the -0 flag is set, but the display and internal data will still be in 1-based closed coordinates. If extracting loci from unspecified or uncertain sources, make sure to confirm the coordinate system! The 1-based coordinates as displayed in apb also match the behaviour of both IGV and jbrowse.

If you are using apb to compare to data as seen in htslib bam1_t and bam_pileup1_t structs or equivalent (e.g. pysam alignment records), keep in mind that all coordinate data will be shifted by 1, including qpos.

macOS Terminal App Users

All Alt-based keybinds (e.g. M-b, M-f, M-d) require the terminal to send Option as Meta. In Terminal, enable “Use Option as Meta key” under Settings > Profiles > Keyboard.

Platform Keybind Differences

Some keybindings depend on your keyboard and terminal.

On a mac laptop keyboard, use fn-Left and fn-Right for Home and End, fn-Up and fn-Down for Page Up and Page Down. macOS swallows Ctrl-Left and Ctrl-Right by default, so these may not reach apb. If so, use Option-Left and Option-Right, or Option-b and Option-f, to move by word instead.

On Linux and Windows terminals, most keybinds generally work without extra configuration.

Esc Misbehaviour

Users of multiplexers (screen, tmux) or any software which attempts to catch Esc and use it as the Alt- modifier for a subsequent keypress could plausibly experience the Esc key behaving unreliably in apb. If this occurs, you may want to configure the appropriate escape key timeout in your multiplexer. Similar misbehaviour could also plausibly occur when submitting complex queries on very large pileups where the query takes a non-trivial period of time (max ~1s) to be processed - pressing ESC followed by a subsequent keypress could be interpreted as Alt-Key. In this case just press Esc again and ensure that the focus has switched before continuing typing. Neither of these issues have been observed in use so far; both are theoretical.

Fonts

Your experience of apb, and many other terminal-based applications, may be improved by installing a modern terminal font designed for legibility. apb recommends 0xProto (https://github.com/0xType/0xProto) try it (and others) out at https://www.programmingfonts.org/#oxproto.


See the README or the project GitHub (https://github.com/blex-max/apb-pileup-browser) for installation instructions, the project roadmap, and other background.