apb Manual - A Pileup Browser

v0.4.0

This is the manual for apb version 0.4.0. This manual is generated directly from the apb binary. If you need a different version of the manual, please check the helptext for instructions as to how to generate the manual for your version of the tool.

If you are reading this manual from a file on disk (rather than the webpage hosted with the apb GitHub repo), it is recommended to read this manual using a markdown viewer. If you are reading on a cluster and do not have a markdown viewer, try a pager such as less. If you have a sufficiently recent version of less, you can use less --wordwrap for an improved experience.

apb is subject to change pending user feedback until the 1.0 release. Some aspects of the tool are unpolished and quirky - I will continue to work on those. In the meantime please send any feedback via GitHub issues at https://github.com/blex-max/apb-pileup-browser, where you will also find the README for installation instructions, the project roadmap, and other background.

1) Overview

Given one or more alignment files and a genomic locus apb retrieves all reads aligned to that locus and loads the reads into a fast, queryable database. The TUI then displays the aligned reads alongside user-selected data for each read (e.g. mapping quality, leftmost alignment position, etc.), and provides a command line at which you can enter commands to query the reads or change the display. The display and command line can be navigated via simple arrow key navigation.

Important

apb displays all coordinate data in 1-based closed coordinates. This includes the LOCUS argument passed at the command line.

2) CLI Usage

See the helptext from apb --help for the canonical guide to the CLI. Some additional context is given here.

apb LOCUS FILE... opens one or more alignment files (SAM/BAM/CRAM) at a locus (chr1:12345), loads the pileup, and launches the TUI. A locus is specified in the form contig:coordinate - only a single coordinate needs to be provided, rather than a length-1 range as in many samtools commands. If a reference is provided via --ref PATH, the reference will be shown and the alignment view will be enriched to indicate divergence from reference in the displayed reads.

apb --demo runs against synthetic data generated by apb and does not require any file input. It is useful for learning apb, but note that since the data is artifically generated it is somewhat unrealistic.

apb --db DB reopens a database file previously produced by --dump (or the in-TUI dump command), and does not take any other file input.

3) TUI Usage

On starting apb you will enter the TUI. The TUI is split into 3 major regions. On the left you have an alignment viewer; on the right you have a table of per-read data. Information about the pileup locus is displayed at the top left of the alignment viewer. At the bottom you have a command line at which you can enter commands to query the reads and control the display. Commands are typed at the command promt >, and return messages are displayed below at the double chevron. Queries will be displayed at the top right of the command line, including the count of reads matching the query. In the alignment pane, = matches the reference, x is a deletion, and a ring over a base marks an insertion, displayed beneath at ^.

3.1) Basic Navigation

The alignment view (i.e. the reads) are scrolled up/down with the arrow keys. Mouse scroll may also work if your terminal is configured to send mouse scrolling as arrow keys, which is reasonably common. Normal typing goes directly to the command line at the bottom of the TUI. Enter dispatches the command. This reference is also available via help nav at the in-app command line.

 M-: Alt | C-: Ctrl | S-: Shift                     
                                                    
 BROWSER PANE                                       
  Up / Down         scroll one row                  
  S-Left / S-Right  pan alignment view side-to-side 
  PgUp / PgDn       scroll one page                 
                                                    
 COMMAND LINE (readline-based)                      
  Enter             submit command                  
  Left / Right      move cursor                     
  C-Left / C-Right                                  
  M-Left / M-Right                                  
  M-b / M-f         back / forward one word         
  C-a / C-e                                         
  Home / End        start / end of line             
  Bksp / Del        delete char back / forward      
  M-Bksp / M-d      delete word back / forward      
  C-k / C-u         delete to end / start of line   
  C-p / C-n         scroll command history          
  C-c               clear input, else quit apb      
                                                    
 QUERY MANAGEMENT                                   
  C-q               recall current query to command 
                    line for editing.               
  C-w               recall where clause of current  
                    query to command line.          
  C-o               recall order by clause of       
                    current query to command line.  
  C-/               undo last modification to       
                    query.                          

Ctrl-C clears the command line if any input is present, and exits the program otherwise.

3.2) Command Reference

Every command submitted at the command line is interpreted like <command> [args]. Shorthand aliases are provided for many commands for quick use. This reference is built directly from the command implementation in src and is also available via help cmd at the in-app command line.

 COMMAND REFERENCE
  `help [(nav|cmd|col)]`:
    Show help for given topic,
    or general help with no args.
     aliases: h, ?

  `where [condition]`:
    Start a new WHERE clause, overwriting
    any existing condition. Omitting the
    [condition] argument will clear the current
    WHERE clause. For a reference of queryable
    columns, check the manual or run `? col`.
     aliases: w, wh, WHERE

  `order by [condition]`:
    Sort reads by ORDER BY expression. Clears order
    by with no args, resetting to default ordering.
    When viewing multiple alignments, reads are
    always sorted by file as the primary key.
     aliases: o, order, ORDER BY, sort

  `count <condition>`:
    Count reads matching <condition>, AND-concatenated
    onto the current WHERE clause if present.
     aliases: ct

  `count all <condition>`:
    Count all reads matching <condition>
     aliases: cta

  `clear `:
    Clear current query, resetting
    both WHERE and ORDER BY.
     aliases: cl

  `col <field-name>...`:
    Show/hide read data columns in the
    table pane. For a list of available
    columns, check the manual or run `? col`.
     aliases: c

  `toggle (quality|insertion|table|sparkline)`:
    Toggle various aspects of the display.
    - quality:   read quality track in
                 alignment pane.
    - insertion: read insertion track in
                 alignment pane.
    - table:     browser table pane.
    - sparkline: switch quality track
                 between Phred ASCII and a
                 quick-look sparkline graph.
    `toggle` accepts any unambiguous substring
    of the argument, e.g. `tg qual`.
     aliases: tg

  `dump <path>`:
    Write the pileup database to an sqlite3 dump file.

3.3) Column Reference

For each read, the database stores the columns detailed below. All columns are queryable in where/order by/count commands. If the content of a column is not clearly displayed by the alignment view, the column can be displayed alongside the reads in tabular format. This reference table is also available via help col at the in-app command line.


 TABLE REFERENCE                                
                                                
 The following columns are available for        
 querying. `qname` through `mstart` may be      
 displayed in the table pane with the `col`     
 command.                                       
                                                
  `path`:                                       
      text / NULL                               
      Path to alignment file.                   
  `seq`:                                        
      text / NULL                               
      the read sequence.                        
  `qual`:                                       
      text / NULL                               
      the read Phred base qualities.            
  `base`:                                       
      character / NULL                          
      Nucleotide at the pileup position. NULL   
      if base is deleted, or seq is missing.    
  `indel`:                                      
      integer                                   
      indel length to the next mapped           
      nucleotide in the read.                   
      (0 = no indel, >0 insertion, <0 deletion) 
  `qname`:                                      
      text / NULL                               
      read/template name.                       
  `flag`:                                       
      integer (bitfield)                        
      SAM bitwise FLAG.                         
  `start`:                                      
      integer > 0                               
      1-based leftmost mapping position of      
      read.                                     
  `end`:                                        
      integer >= start                          
      1-based rightmost mapping position of     
      read.                                     
  `mapq`:                                       
      integer 0-255                             
      read mapping quality.                     
  `basequal`:                                   
      integer / NULL                            
      Phred base quality at the pileup          
      position. NULL if base is deleted, or     
      qual is missing.                          
  `qpos`:                                       
      integer > 0                               
      1-based offset into `seq`/`qual` to       
      pileup locus position. In deletions, this 
      is the first base after the deletion.     
  `cigar`:                                      
      text                                      
      CIGAR string of alignment operations.     
  `mtid`:                                       
      text / NULL                               
      reference name of the mate/next read.     
      = if mate is on same contig. Usually = if 
      mate is unmapped. Usually NULL if read    
      unpaired.                                 
      (use `flag` to check if mate is unmapped) 
  `mstart`:                                     
      integer > 0 / NULL                        
      mate/next read's leftmost mapping         
      position. Usually identical to `start`    
      if mate is unmapped. Usually NULL if read 
      unpaired.                                 
      (use `flag` to check if mate is unmapped) 
  `tags`:                                       
      text (JSON) / NULL                        
      aux tags as JSON; able to be individually 
      queried using ->> operator. NULL if tags  
      are missing.                              
  `is_del`:                                     
      integer 1-0 (true/false)                  
      1 if `base` is a deletion.                
  `is_head`:                                    
      integer 1-0 (true/false)                  
      1 if `qpos` is the read's first aligned   
      base.                                     
  `is_tail`:                                    
      integer 1-0 (true/false)                  
      1 if `qpos` is the read's last aligned    
      base.                                     
  `is_refskip`:                                 
      integer 1-0 (true/false)                  
      1 if `qpos` is a reference skip.          

For a guide to the symbols used in the quality string, see https://en.wikipedia.org/wiki/Phred_quality_score#Symbols. Note that toggle sparkline switches the quality track in the browser pane between ASCII display of quality scores, and a sparkline graph - i.e. FAFF-FF<FFI vs ▆▅▆▆▁▆▆▄▆▆▆.

indel may require further explanation. Essentially, if the base at the pileup position is followed by an indel, then indel will contain the size of that indel event. A deletion is represented by a negative size (bases lost), and an insertion is represented by a positive size (bases gained).

For advanced users, note that most of these map directly onto fields in htslib’s bam_pileup1_t and bam1_t structs - consult the schema.

3.4) Querying the Pileup

The apb query commands provide a simple wrapper around (sqlite3-based) SQL. This means you can directly query the pileup to view and count reads by their properites, such as the base, flag, and cigar string. apb provides three major commands for querying - where, for filtering the pileup to reads matching a condition; order by, for sorting or grouping reads in the view; and count, for quantifying reads. This is best explained by example:

If this is your first time using apb, you may find it useful to walk through the following example in the demo mode, which can be accessed at the CLI by apb --demo. The numbers will not exactly match the examples as the examples are only illustrative.

Imagine we are inspecting a candidate SNV variant locus. We may start by filtering the pileup to only reads which the caller should consider. Specifics will vary by caller, but a common starting point is “reads which map well, have mapped mates, and aren’t duplicates, qc fails, or secondary/supplementary alignments”. At the command line we enter:

where mapq > 0 AND (flag & 3848) = 0;

This is an SQL WHERE clause - essentially a filter. The browser will update to show only reads matching that filter, and the status line will now show:

- WHERE mapq > 0 AND (flag & 3848) = 0: 76/100 reads

Let’s say our candidate SNV is A->T. We can sort the reads to group reads expressing the variant base at the pileup locus and view them in the alignment pane:

order by base IS 'T' DESC

This is an SQL ORDER BY clause - essentially a sort. DESC is simply the SQL keyword for descending order (ASC is for ascending, which is the default). DESC puts reads that match our condition to the top of the sort for each alignment file in the pileup. The status line will now show:

- WHERE mapq > 0 AND (flag & 3848) = 0: 76/100 reads │ ORDER BY base IS 'T' DESC

The alignment pane now shows only reads matching our WHERE condition, grouped by our ORDER BY condition. We may now explore the pileup visually for any properties of interest, e.g. if most of the reads supporting the variant are soft-clipped.

After exploring the pileup, it can be useful to quantify the locus in more detail. For example, to count the number of reads exhibiting the variant:

count base IS 'T'

This will return a message below the command line:

- base IS 'T' / mapq > 0 AND (flag & 3848) = 0: 24/76 reads

The count returned is the count of reads matching the current WHERE condition AND the condition provided to count, against the total count of reads matching the current WHERE condition. If instead you wanted to count all reads matching a condition and ignoring the current WHERE condition, use count all [condition]:

count all base IS 'T'

returning:

- base IS 'T': 28/100 reads

From this we may conclude that four reads exhibit the variant, but are filtered out by our current WHERE clause on mapq and flag. We may consider relaxing our requirement that the mate must be mapped to see if this brings any of those four reads into view. To do so, we can simply grab the current WHERE clause from the status bar with Ctrl-w. This places it in the command line, where we may edit and resubmit:

where mapq > 0 AND (flag & 3840 /* edited from 3848 */ ) = 0;

Editing the query is made easy by using the readline-style editing commands detailed in Section 3.1. This concludes the example, but is by no means the limit of what you can do with apb. See the next section for further examples and snippets.

Important

flag is a bitmask of read properties defined by the SAM format - see https://broadinstitute.github.io/picard/explain-flags.html from the Broad Institute, which both explains the meaning of the flag bits and allows you to calcuate the number (like 3848) which corresponds to a particular set of properties. See https://www.w3schools.com/programming/prog_operators_bitwise.php for a quick introduction to bitmask syntax. SQLite supports the bitwise and, or, and not.

Important

In SQL the ORDER BY clause can take multiple keys. This is supported by the order by command, so we could for example have written order by base IS 'T' DESC, basequal. This would have sorted the view such that reads exhibiting a ‘T’ at the pileup locus will appear first, but now within that group, reads will be sorted by the quality of that base, in ascending order (lowest first).

Important

Missing values are encoded as NULL, and different operators behave differently with NULL values in SQL. For example, a deleted base is NULL - where base != 'A' will exclude deletions, as NULL always evaluates to false in comparisons using !=, i.e. NULL != 'A' is false. Whereas, where base IS NOT 'A' will include deletions as NULL IS NOT 'A' evaluates to true. To check for the presence of missing values for a field, use where <col> IS NULL.

Important

In SQL, AND conditions always bind more tightly than OR. This means that:

WHERE a AND b AND c OR d

is equivalent to:

WHERE (a AND b AND c) OR d

If you want to bind two conditions together, use brackets:

WHERE a AND b AND (c OR d)

3.5) Further Querying Examples

Querying is flexible and powerful. Some useful examples:

Querying by Path

When viewing multiple alignment pileups, use instr() to subset your query to a specific file, e.g.:

where instr(path, 'PD12345a') and base != 'A'

Indels at the Pileup Locus

This is made quite simple by the indel column. For example, to search for an 3-base insertion anchored at the pileup:

where indel = 3

deletions are represented by negative numbers, so for a 3-base deletion:

where indel = -3

Note that these commands search for indels anchored at the pileup base, i.e. beginning immediately after the pileup locus. If you want to search for deleted bases at the pileup locus itself, use:

where is_del

Aux Tags

Aux tags may be queried with ->>. For example, to subset a query to a read group via the RG tag:

where base != 'G' and tags ->> '$.RG' = 'sample1'

For numeric tags, all comparison an arithmetic operators are available:

where tags ->> '$.NM' > 2

A read missing a tag returns null, so:

where tags ->> '$.RG' is null

Would find reads missing the RG tag.

CIGAR Querying

Pattern matching is available on all text fields, such as the CIGAR string. E.g. to filter for reads containing a 3-base insertion at any position:

where instr(cigar, '3I')

or for reads containing soft clipping of any length:

where instr(cigar, 'S')

Read Length and Arithmetic

Arithmetic functions are available. For example:

where end - start < 100

would filter the view to reads less than 100 bases long.

Motifs at the Query Position

This is a slightly more advanced example. qpos is the 1-based offset into seq for the base at the pileup position, matching SQLite’s own 1-based substr() indexing, so to search for the 4-mer GATC starting at the query position:

where substr(seq, qpos, 4) = 'GATC'

LIKE (_/%) and GLOB (?/*/[ACG]) both work as wildcards. To search for two possible trinucleotide motifs at the query position:

where substr(seq, qpos, 3) glob 'A[CG]T'

You can also search for motifs within a window of the seq string. This command searches for GATC within the first 10 bases of the read:

where instr(substr(seq, 1, 10), 'GATC') > 0

Any and all of these approaches may be combined, and more is possible. See SQLite’s expression/function reference - https://sqlite.org/lang_expr.html.


Addenda

The dump Functionality

A dump is a small, self-contained sqlite3 file with just the reads at the specified locus. A few use cases:

Note that all coordinate data within the dump is 1-based, matching the display.

A Word on Indexing Systems

htslib/samtools/bcftools, and by extension all alignment and VCF data, mix 0-based half-open and 1-based closed coordinate systems. This can be tricky to navigate.

apb uses 1-based closed coordinates throughtout, i.e coordinates start at 1, and coordinate ranges are [x-y] where both x and y are included. 1-based closed coordinates match the human-readable VCF and SAM format specs, and to samtools commands e.g. samtools view .... Coordinates from these sources, i.e. a variant locus from a VCF, may be passed directly to the locus CLI argument at startup. Loci from 0-indexed sources, e.g. BED files, may be passed directly to the CLI argument if the -0 flag is set, but the display and internal data will still be in 1-based closed coordinates. If extracting loci from unspecified or uncertain sources, make sure to confirm the coordinate system! The 1-based coordinates as displayed in apb also match the behaviour of both IGV and jbrowse.

If you are using apb to compare to data as seen in htslib bam1_t and bam_pileup1_t structs or equivalent (e.g. pysam alignment records), keep in mind that all coordinate data will be shifted by 1, including qpos.

macOS Terminal App Users

The Alt-based command line keybinds (M-b, M-f, M-d and M-Bksp) require the terminal to send Option as Meta. In Terminal, enable “Use Option as Meta key” under Settings > Profiles > Keyboard.

Platform Keybind Differences

Some command line and navigation keys depend on your keyboard and terminal.

On a mac laptop keyboard, which has no dedicated Home, End or forward Delete keys, use fn+Left and fn+Right for Home and End, and fn+Delete for forward Delete. Ctrl-A and Ctrl-E also move to the start and end of the line. macOS uses Ctrl-Left and Ctrl-Right to switch Spaces, so these may not reach apb. Use Option-Left and Option-Right, or Alt-b and Alt-f, to move by word instead. PgUp and PgDn can be sent using the fn modifier key with the up and down arrow keys if using a mac laptop keyboard.

On Linux and Windows terminals, Alt-based keybinds generally work without extra configuration.

Fonts

Your experience of apb, and many other terminal-based applications, may be improved by installing a modern terminal font designed for legibility. apb recommends 0xProto (https://github.com/0xType/0xProto) try it (and others) out at https://www.programmingfonts.org/#oxproto.


See the README or the project GitHub (https://github.com/blex-max/apb-pileup-browser) for installation instructions, the project roadmap, and other background.