v0.5.0
This is the manual for apb version 0.5.0. This manual is generated directly from the apb binary. If you need a different version of the manual, please check the helptext for instructions as to how to generate the manual for your version of the tool.
If you are reading this manual from a file on disk (rather than the
webpage hosted with the apb GitHub repo), it is recommended to read this
manual using a markdown viewer. If you are reading on a cluster and do
not have a markdown viewer, try a pager such as less. If
you have a sufficiently recent version of less, you can use
less --wordwrap for an improved experience.
apb is subject to change pending user feedback until the 1.0 release. Some aspects of the tool are unpolished and quirky - I will continue to work on those. In the meantime please send any feedback via GitHub issues at https://github.com/blex-max/apb-pileup-browser, where you will also find the README for installation instructions, the project roadmap, and other background.
Given one or more alignment files and a genomic locus apb retrieves all reads aligned to that locus and loads the reads into a fast, queryable database. The TUI then displays the aligned reads alongside user-selected data for each read (e.g. mapping quality, leftmost alignment position, etc.), and provides a command prompt at which you can enter commands to query the reads or change the display. The display and command prompt can be navigated via simple arrow key navigation.
apb displays all coordinate data in 1-based closed coordinates. This includes the LOCUS argument passed at startup to the CLI.
See the helptext from apb --help for the canonical guide
to the CLI. Some additional context is given here.
apb LOCUS FILE... opens one or more alignment files
(SAM/BAM/CRAM) at a locus (chr1:12345), loads the pileup,
and launches the TUI. A locus is specified in the form
contig:coordinate - only a single coordinate needs to be
provided, rather than a length-1 range as in many samtools
commands. If a reference is provided via --ref PATH, the
reference will be shown and the alignment pane will be enriched to
indicate divergence from reference in the displayed reads.
apb --demo runs against synthetic data generated by apb
and does not require any file input. It is useful for learning apb, but
note that since the data is artifically generated it is somewhat
unrealistic.
apb --db DB reopens a database file previously produced
by --dump (or the in-TUI dump command), and
does not take any other file input.
On starting apb you will enter the TUI. The TUI is split into 2 major
regions. At the top you have the read viewer, made up of an alignment
pane on the left and a table pane of per-read data on the right.
Information about the pileup locus is displayed at the top left of the
read viewer. At the bottom you have a command prompt at which you can
enter commands to query the reads and control the display. Commands are
typed at the command prompt >, and return messages are
displayed below at the double chevron. Queries will be displayed at the
top right of the prompt, including the count of reads matching the
query. In the alignment pane, = matches the reference,
x is a deletion, and a ring over a base marks an insertion,
displayed beneath at ^.
apb has a modal interface, which means that the way the program
responds to your input depends on which mode you are in. There are only
two major modes: viewer, for viewing the reads in the read
viewer, and prompt, for controlling the command prompt. You
can switch between modes with the Esc key, and the current
mode is always beneath the read viewer as
focused (Esc): <mode>.
In viewer mode, the read viewer is scrolled, and the alignment pane
panned, with the arrow keys. Mouse scroll may also work if your terminal
is configured to send mouse scrolling as arrow keys, which is reasonably
common. In prompt mode, normal typing goes directly to the prompt at the
>, and Enter dispatches a typed command.
Keyboard shortcuts are provided for easily navigating and editing text
while in prompt mode.
The following reference table is also available via
help nav at the prompt:
M-: Alt | C-: Ctrl
ALWAYS ACTIVE:
Esc change focus/mode between the
read viewer and the prompt
C-c quit apb
FOCUS: VIEWER
Up / Down scroll one row
Left / Right pan alignment pane side-to-side
PgUp / PgDn scroll one page
FOCUS: PROMPT
C-w recall where clause to command
line
C-o recall order by clause to
command line
C-/ undo last modification to
query
Enter submit command
Left / Right move cursor
Up / Down scroll command history
C-Left / C-Right
M-Left / M-Right
M-b / M-f back / forward one word
C-a / C-e
Home / End start / end of line
Bksp / C-d delete char back / forward
M-Bksp / M-d delete word back / forward
C-k / C-u delete to end / start of line
Every command submitted at the prompt is interpreted like
<command> [args]. Shorthand aliases are provided for
many commands for quick use.
The following reference table is also available via
help cmd at the prompt.
COMMAND REFERENCE
`help [(nav|cmd|col)]`:
Show help for given topic,
or general help with no args.
aliases: h, ?
`quit `:
Exit apb.
aliases: q, exit, close
`where [condition]`:
Start a new WHERE clause, overwriting
any existing condition. Omitting the
[condition] argument will clear the current
WHERE clause. For a reference of queryable
columns, check the manual or run `? col`.
aliases: w, wh, WHERE
`order by [condition]`:
Sort reads by ORDER BY expression. Clears order
by with no args, resetting to default ordering.
When viewing multiple alignments, reads are
always sorted by file as the primary key.
aliases: o, order, ORDER BY, sort
`count <condition>`:
Count reads matching <condition>, AND-concatenated
onto the current WHERE clause if present.
aliases: ct
`count all <condition>`:
Count all reads matching <condition>
aliases: cta
`clear `:
Clear current query, resetting
both WHERE and ORDER BY.
aliases: cl
`col <field-name>...`:
Show/hide read data columns in the
table pane. For a list of available
columns, check the manual or run `? col`.
aliases: c
`toggle (quality|insertion|table|sparkline)`:
Toggle various aspects of the display.
- quality: read quality track in
alignment pane.
- insertion: read insertion track in
alignment pane.
- table: table pane.
- sparkline: switch quality track
between Phred ASCII and a
quick-look sparkline graph.
`toggle` accepts any unambiguous substring
of the argument, e.g. `tg qual`.
aliases: tg
`dump <path>`:
Write the pileup database to an sqlite3 dump file.
For each read, the database stores the columns detailed below. All
columns are queryable in
where/order by/count commands. If
the content of a column is not clearly displayed by the alignment pane,
the column can be displayed in the table pane.
The following reference table is also available via
help col at the prompt.
TABLE REFERENCE
The following columns are available for
querying. `qname` through `mstart` may be
displayed in the table pane with the `col`
command.
`path`:
text
Path to alignment file.
`seq`:
text / NULL
the read sequence. NULL if missing.
`qual`:
text / NULL
the read Phred base qualities. NULL if
missing.
`base`:
character / NULL
Nucleotide at the pileup position. NULL
if base is deleted, or `seq` is missing.
`indel`:
integer
indel length to the next mapped
nucleotide in the read.
(0 = no indel, >0 insertion, <0 deletion)
`qname`:
text / NULL
the read/template name.
NULL if missing.
`flag`:
integer (bitfield)
SAM bitwise FLAG.
`start`:
integer > 0
1-based leftmost mapping position of
read.
`end`:
integer >= start
1-based rightmost mapping position of
read.
`mapq`:
integer 0-255
read mapping quality.
`basequal`:
integer / NULL
Phred base quality at the pileup
position. NULL if base is deleted, or
`qual` is missing.
`qpos`:
integer > 0
1-based offset into `seq`/`qual` to
pileup locus position. In deletions, this
is the first base after the deletion.
`cigar`:
text
CIGAR string of alignment operations.
`mtid`:
text / NULL
reference name of the mate/next read.
= if mate is on same contig. Usually = if
mate is unmapped. Usually NULL if read
unpaired.
(use `flag` to check if mate is unmapped)
`mstart`:
integer > 0 / NULL
mate/next read's leftmost mapping
position. Usually identical to `start`
if mate is unmapped. Usually NULL if read
unpaired.
(use `flag` to check if mate is unmapped)
`tags`:
text (JSON) / NULL
aux tags as JSON; able to be individually
queried using ->> operator. NULL if tags
are missing.
`is_del`:
integer 1-0 (true/false)
1 if `base` is a deletion.
`is_head`:
integer 1-0 (true/false)
1 if `qpos` is the read's first aligned
base.
`is_tail`:
integer 1-0 (true/false)
1 if `qpos` is the read's last aligned
base.
`is_refskip`:
integer 1-0 (true/false)
1 if `qpos` is a reference skip.
For a guide to the symbols used in the quality string, see https://en.wikipedia.org/wiki/Phred_quality_score#Symbols.
Note that toggle sparkline switches the quality track in
the alignment pane between ASCII display of quality scores, and a
sparkline graph - i.e. FAFF-FF<FFI vs
▆▅▆▆▁▆▆▄▆▆▆.
indel may require further explanation. Essentially, if
the base at the pileup position is followed by an indel, then
indel will contain the size of that indel event. A deletion
is represented by a negative size (bases lost), and an insertion is
represented by a positive size (bases gained).
For advanced users, note that most of these map directly onto fields
in htslib’s bam_pileup1_t and bam1_t structs -
consult the schema.
The apb query commands provide a simple wrapper around
(sqlite3-based) SQL. This means you can directly query the pileup to
view and count reads by their properties, such as the base, flag, and
cigar string. apb provides three major commands for querying -
where, for filtering the pileup to reads matching a
condition; order by, for sorting or grouping reads in the
view; and count, for quantifying reads. This is best
explained by example:
If this is your first time using apb, you may find it useful to walk
through the following example in the demo mode, which can be accessed at
the CLI by apb --demo. The numbers will not exactly match
the examples as the examples are only illustrative.
Imagine we are inspecting a candidate SNV variant locus. We may start by filtering the pileup to only reads which the caller should consider. Specifics will vary by caller, but a common starting point is “reads which map well, have mapped mates, and aren’t duplicates, qc fails, or secondary/supplementary alignments”. At the command prompt we enter:
where mapq > 0 AND (flag & 3848) = 0;
This is an SQL WHERE clause - essentially a filter. The read viewer will update to show only reads matching that filter, and the status line will now show:
- WHERE mapq > 0 AND (flag & 3848) = 0: 76/100 reads
Let’s say our candidate SNV is A->T. We can sort the reads to group reads expressing the variant base at the pileup locus and view them in the read viewer:
order by base IS 'T' DESC
This is an SQL ORDER BY clause - essentially a sort. DESC is simply the SQL keyword for descending order (ASC is for ascending, which is the default). DESC puts reads that match our condition to the top of the sort for each alignment file in the pileup. The status line will now show:
- WHERE mapq > 0 AND (flag & 3848) = 0: 76/100 reads │ ORDER BY base IS 'T' DESC
The read viewer now shows only reads matching our WHERE condition,
grouped by our ORDER BY condition. We may now explore the pileup
visually for any properties of interest, e.g. if most of the reads
supporting the variant are soft-clipped. Switch to the read viewer with
Esc, and switch back to the prompt with Esc
again when finished.
After exploring the pileup, it can be useful to quantify the locus in more detail. For example, to count the number of reads exhibiting the variant:
count base IS 'T'
This will return a message below the prompt at »:
- base IS 'T' / mapq > 0 AND (flag & 3848) = 0: 24/76 reads
The count returned is the count of reads matching the current WHERE
condition AND the condition provided to count, against the total count
of reads matching the current WHERE condition. If instead you wanted to
count all reads matching a condition and ignoring the current WHERE
condition, use count all [condition]:
count all base IS 'T'
returning:
- base IS 'T': 28/100 reads
From this we may conclude that four reads exhibit the variant, but
are filtered out by our current WHERE clause on mapq and
flag. We may consider relaxing our requirement that the
mate must be mapped to see if this brings any of those four reads into
view. To do so, we can simply grab the current WHERE clause from the
status bar with Ctrl-w. This recalls it to the prompt,
where we may edit and resubmit:
where mapq > 0 AND (flag & 3840 /* edited from 3848 */ ) = 0;
Editing the query is made easy by using the editing key shortcuts detailed in Section 3.1. This concludes the example, but is by no means the limit of what you can do with apb. See the next section for further examples and snippets.
flag is a bitmask of read properties defined by the SAM
format - see https://broadinstitute.github.io/picard/explain-flags.html
from the Broad Institute, which both explains the meaning of the flag
bits and allows you to calcuate the number (like 3848) which corresponds
to a particular set of properties. See https://www.w3schools.com/programming/prog_operators_bitwise.php
for a quick introduction to bitmask syntax. SQLite supports the bitwise
and, or, and not.
In SQL the ORDER BY clause can take multiple keys. This
is supported by the order by command, so we could for
example have written order by base IS 'T' DESC, basequal.
This would have sorted the view such that reads exhibiting a ‘T’ at the
pileup locus will appear first, but now within that group, reads will be
sorted by the quality of that base, in ascending order (lowest
first).
Missing values are encoded as NULL, and different operators behave
differently with NULL values in SQL. For example, a deleted
base is NULL - where base != 'A' will exclude
deletions, as NULL always evaluates to false in comparisons using
!=, i.e. NULL != 'A' is false. Whereas,
where base IS NOT 'A' will include deletions as
NULL IS NOT 'A' evaluates to true. To check for the
presence of missing values for a field, use
where <col> IS NULL.
In SQL, AND conditions always bind more tightly than OR. This means that:
WHERE a AND b AND c OR d
is equivalent to:
WHERE (a AND b AND c) OR d
If you want to bind two conditions together, use brackets:
WHERE a AND b AND (c OR d)
Querying is flexible and powerful. Some useful examples:
When viewing multiple alignment pileups, use instr() to
subset your query to a specific file, e.g.:
where instr(path, 'PD12345a') and base != 'A'
This is made quite simple by the indel column. For
example, to search for a 3-base insertion anchored at the pileup:
where indel = 3
deletions are represented by negative numbers, so for a 3-base deletion:
where indel = -3
Note that these commands search for indels anchored at the pileup base, i.e. beginning immediately after the pileup locus. If you want to search for deleted bases at the pileup locus itself, use:
where is_del
Aux tags may be queried with ->>. For example, to
subset a query to a read group via the RG tag:
where base != 'G' and tags ->> '$.RG' = 'sample1'
For numeric tags, all comparison an arithmetic operators are available:
where tags ->> '$.NM' > 2
A read missing a tag returns null, so:
where tags ->> '$.RG' is null
Would find reads missing the RG tag.
Pattern matching is available on all text fields, such as the CIGAR string. E.g. to filter for reads containing a 3-base insertion at any position:
where instr(cigar, '3I')
or for reads containing soft clipping of any length:
where instr(cigar, 'S')
Arithmetic functions are available. For example:
where end - start < 100
would filter the view to reads less than 100 bases long.
This is a slightly more advanced example. qpos is the
1-based offset into seq for the base at the pileup
position, matching SQLite’s own 1-based substr() indexing,
so to search for the 4-mer GATC starting at the query
position:
where substr(seq, qpos, 4) = 'GATC'
LIKE (_/%) and
GLOB (?/*/[ACG])
both work as wildcards. To search for two possible trinucleotide motifs
at the query position:
where substr(seq, qpos, 3) glob 'A[CG]T'
You can also search for motifs within a window of the
seq string. This command searches for GATC
within the first 10 bases of the read:
where instr(substr(seq, 1, 10), 'GATC') > 0
Any and all of these approaches may be combined, and more is possible. See SQLite’s expression/function reference - https://sqlite.org/lang_expr.html.
dump FunctionalityA dump is a small, self-contained sqlite3 file with just the reads at the specified locus. A few use cases:
sqlite3 my.db allows for complex querying
(GROUP BY, aggregates, summaries, etc.) and programmatic
use of apb
(apb --dump my.db --ref my.ref chr1:12345 my.bam && sqlite3 -header -csv my.db < query.sql > results.csv).apb if they’re happy just to use
sqlite3).Note that all coordinate data within the dump is 1-based, matching the display.
htslib/samtools/bcftools, and by extension all alignment and VCF data, mix 0-based half-open and 1-based closed coordinate systems. This can be tricky to navigate.
apb uses 1-based closed coordinates throughtout, i.e coordinates
start at 1, and coordinate ranges are [x-y] where both x and y are
included. 1-based closed coordinates match the human-readable VCF and
SAM format specs, and to samtools commands
e.g. samtools view .... Coordinates from these sources,
i.e. a variant locus from a VCF, may be passed directly to the locus CLI
argument at startup. Loci from 0-indexed sources, e.g. BED files, may be
passed directly to the CLI argument if the -0 flag is set,
but the display and internal data will still be in 1-based closed
coordinates. If extracting loci from unspecified or uncertain sources,
make sure to confirm the coordinate system! The 1-based coordinates as
displayed in apb also match the behaviour of both IGV and
jbrowse.
If you are using apb to compare to data as seen in htslib
bam1_t and bam_pileup1_t structs or equivalent
(e.g. pysam alignment records), keep in mind that all coordinate data
will be shifted by 1, including qpos.
All Alt-based keybinds (e.g. M-b, M-f, M-d) require the terminal to send Option as Meta. In Terminal, enable “Use Option as Meta key” under Settings > Profiles > Keyboard.
Some keybindings depend on your keyboard and terminal.
On a mac laptop keyboard, use fn-Left and
fn-Right for Home and End, fn-Up and
fn-Down for Page Up and Page Down. macOS swallows
Ctrl-Left and Ctrl-Right by default, so these
may not reach apb. If so, use Option-Left and
Option-Right, or Option-b and
Option-f, to move by word instead.
On Linux and Windows terminals, most keybinds generally work without extra configuration.
Esc MisbehaviourUsers of multiplexers (screen, tmux) or any software which attempts
to catch Esc and use it as the Alt- modifier for a
subsequent keypress could plausibly experience the Esc key
behaving unreliably in apb. If this occurs, you may want to configure
the appropriate escape key timeout in your multiplexer. Similar
misbehaviour could also plausibly occur when submitting complex queries
on very large pileups where the query takes a non-trivial period of time
(max ~1s) to be processed - pressing ESC followed by a subsequent
keypress could be interpreted as Alt-Key. In this case just press Esc
again and ensure that the focus has switched before continuing typing.
Neither of these issues have been observed in use so far; both are
theoretical.
Your experience of apb, and many other terminal-based applications,
may be improved by installing a modern terminal font designed for
legibility. apb recommends 0xProto (https://github.com/0xType/0xProto) try it (and others)
out at https://www.programmingfonts.org/#oxproto.
See the README or the project GitHub (https://github.com/blex-max/apb-pileup-browser) for installation instructions, the project roadmap, and other background.