v0.4.0
This is the manual for apb version 0.4.0. This manual is generated directly from the apb binary. If you need a different version of the manual, please check the helptext for instructions as to how to generate the manual for your version of the tool.
If you are reading this manual from a file on disk (rather than the
webpage hosted with the apb GitHub repo), it is recommended to read this
manual using a markdown viewer. If you are reading on a cluster and do
not have a markdown viewer, try a pager such as less. If
you have a sufficiently recent version of less, you can use
less --wordwrap for an improved experience.
apb is subject to change pending user feedback until the 1.0 release. Some aspects of the tool are unpolished and quirky - I will continue to work on those. In the meantime please send any feedback via GitHub issues at https://github.com/blex-max/apb-pileup-browser, where you will also find the README for installation instructions, the project roadmap, and other background.
Given one or more alignment files and a genomic locus apb retrieves all reads aligned to that locus and loads the reads into a fast, queryable database. The TUI then displays the aligned reads alongside user-selected data for each read (e.g. mapping quality, leftmost alignment position, etc.), and provides a command line at which you can enter commands to query the reads or change the display. The display and command line can be navigated via simple arrow key navigation.
apb displays all coordinate data in 1-based closed coordinates. This includes the LOCUS argument passed at the command line.
See the helptext from apb --help for the canonical guide
to the CLI. Some additional context is given here.
apb LOCUS FILE... opens one or more alignment files
(SAM/BAM/CRAM) at a locus (chr1:12345), loads the pileup,
and launches the TUI. A locus is specified in the form
contig:coordinate - only a single coordinate needs to be
provided, rather than a length-1 range as in many samtools
commands. If a reference is provided via --ref PATH, the
reference will be shown and the alignment view will be enriched to
indicate divergence from reference in the displayed reads.
apb --demo runs against synthetic data generated by apb
and does not require any file input. It is useful for learning apb, but
note that since the data is artifically generated it is somewhat
unrealistic.
apb --db DB reopens a database file previously produced
by --dump (or the in-TUI dump command), and
does not take any other file input.
On starting apb you will enter the TUI. The TUI is split into 3 major
regions. On the left you have an alignment viewer; on the right you have
a table of per-read data. Information about the pileup locus is
displayed at the top left of the alignment viewer. At the bottom you
have a command line at which you can enter commands to query the reads
and control the display. Commands are typed at the command promt
>, and return messages are displayed below at the double
chevron. Queries will be displayed at the top right of the command line,
including the count of reads matching the query. In the alignment pane,
= matches the reference, x is a deletion, and
a ring over a base marks an insertion, displayed beneath at
^.
The alignment view (i.e. the reads) are scrolled up/down with the
arrow keys. Mouse scroll may also work if your terminal is configured to
send mouse scrolling as arrow keys, which is reasonably common. Normal
typing goes directly to the command line at the bottom of the TUI.
Enter dispatches the command. This reference is also
available via help nav at the in-app command line.
M-: Alt | C-: Ctrl | S-: Shift
BROWSER PANE
Up / Down scroll one row
S-Left / S-Right pan alignment view side-to-side
PgUp / PgDn scroll one page
COMMAND LINE (readline-based)
Enter submit command
Left / Right move cursor
C-Left / C-Right
M-Left / M-Right
M-b / M-f back / forward one word
C-a / C-e
Home / End start / end of line
Bksp / Del delete char back / forward
M-Bksp / M-d delete word back / forward
C-k / C-u delete to end / start of line
C-p / C-n scroll command history
C-c clear input, else quit apb
QUERY MANAGEMENT
C-q recall current query to command
line for editing.
C-w recall where clause of current
query to command line.
C-o recall order by clause of
current query to command line.
C-/ undo last modification to
query.
Ctrl-C clears the command line if any input is present, and exits the program otherwise.
Every command submitted at the command line is interpreted like
<command> [args]. Shorthand aliases are provided for
many commands for quick use. This reference is built directly from the
command implementation in src and is also available via
help cmd at the in-app command line.
COMMAND REFERENCE
`help [(nav|cmd|col)]`:
Show help for given topic,
or general help with no args.
aliases: h, ?
`where [condition]`:
Start a new WHERE clause, overwriting
any existing condition. Omitting the
[condition] argument will clear the current
WHERE clause. For a reference of queryable
columns, check the manual or run `? col`.
aliases: w, wh, WHERE
`order by [condition]`:
Sort reads by ORDER BY expression. Clears order
by with no args, resetting to default ordering.
When viewing multiple alignments, reads are
always sorted by file as the primary key.
aliases: o, order, ORDER BY, sort
`count <condition>`:
Count reads matching <condition>, AND-concatenated
onto the current WHERE clause if present.
aliases: ct
`count all <condition>`:
Count all reads matching <condition>
aliases: cta
`clear `:
Clear current query, resetting
both WHERE and ORDER BY.
aliases: cl
`col <field-name>...`:
Show/hide read data columns in the
table pane. For a list of available
columns, check the manual or run `? col`.
aliases: c
`toggle (quality|insertion|table|sparkline)`:
Toggle various aspects of the display.
- quality: read quality track in
alignment pane.
- insertion: read insertion track in
alignment pane.
- table: browser table pane.
- sparkline: switch quality track
between Phred ASCII and a
quick-look sparkline graph.
`toggle` accepts any unambiguous substring
of the argument, e.g. `tg qual`.
aliases: tg
`dump <path>`:
Write the pileup database to an sqlite3 dump file.
For each read, the database stores the columns detailed below. All
columns are queryable in
where/order by/count commands. If
the content of a column is not clearly displayed by the alignment view,
the column can be displayed alongside the reads in tabular format. This
reference table is also available via help col at the
in-app command line.
TABLE REFERENCE
The following columns are available for
querying. `qname` through `mstart` may be
displayed in the table pane with the `col`
command.
`path`:
text / NULL
Path to alignment file.
`seq`:
text / NULL
the read sequence.
`qual`:
text / NULL
the read Phred base qualities.
`base`:
character / NULL
Nucleotide at the pileup position. NULL
if base is deleted, or seq is missing.
`indel`:
integer
indel length to the next mapped
nucleotide in the read.
(0 = no indel, >0 insertion, <0 deletion)
`qname`:
text / NULL
read/template name.
`flag`:
integer (bitfield)
SAM bitwise FLAG.
`start`:
integer > 0
1-based leftmost mapping position of
read.
`end`:
integer >= start
1-based rightmost mapping position of
read.
`mapq`:
integer 0-255
read mapping quality.
`basequal`:
integer / NULL
Phred base quality at the pileup
position. NULL if base is deleted, or
qual is missing.
`qpos`:
integer > 0
1-based offset into `seq`/`qual` to
pileup locus position. In deletions, this
is the first base after the deletion.
`cigar`:
text
CIGAR string of alignment operations.
`mtid`:
text / NULL
reference name of the mate/next read.
= if mate is on same contig. Usually = if
mate is unmapped. Usually NULL if read
unpaired.
(use `flag` to check if mate is unmapped)
`mstart`:
integer > 0 / NULL
mate/next read's leftmost mapping
position. Usually identical to `start`
if mate is unmapped. Usually NULL if read
unpaired.
(use `flag` to check if mate is unmapped)
`tags`:
text (JSON) / NULL
aux tags as JSON; able to be individually
queried using ->> operator. NULL if tags
are missing.
`is_del`:
integer 1-0 (true/false)
1 if `base` is a deletion.
`is_head`:
integer 1-0 (true/false)
1 if `qpos` is the read's first aligned
base.
`is_tail`:
integer 1-0 (true/false)
1 if `qpos` is the read's last aligned
base.
`is_refskip`:
integer 1-0 (true/false)
1 if `qpos` is a reference skip.
For a guide to the symbols used in the quality string, see https://en.wikipedia.org/wiki/Phred_quality_score#Symbols.
Note that toggle sparkline switches the quality track in
the browser pane between ASCII display of quality scores, and a
sparkline graph - i.e. FAFF-FF<FFI vs
▆▅▆▆▁▆▆▄▆▆▆.
indel may require further explanation. Essentially, if
the base at the pileup position is followed by an indel, then
indel will contain the size of that indel event. A deletion
is represented by a negative size (bases lost), and an insertion is
represented by a positive size (bases gained).
For advanced users, note that most of these map directly onto fields
in htslib’s bam_pileup1_t and bam1_t structs -
consult the schema.
The apb query commands provide a simple wrapper around
(sqlite3-based) SQL. This means you can directly query the pileup to
view and count reads by their properites, such as the base, flag, and
cigar string. apb provides three major commands for querying -
where, for filtering the pileup to reads matching a
condition; order by, for sorting or grouping reads in the
view; and count, for quantifying reads. This is best
explained by example:
If this is your first time using apb, you may find it useful to walk
through the following example in the demo mode, which can be accessed at
the CLI by apb --demo. The numbers will not exactly match
the examples as the examples are only illustrative.
Imagine we are inspecting a candidate SNV variant locus. We may start by filtering the pileup to only reads which the caller should consider. Specifics will vary by caller, but a common starting point is “reads which map well, have mapped mates, and aren’t duplicates, qc fails, or secondary/supplementary alignments”. At the command line we enter:
where mapq > 0 AND (flag & 3848) = 0;
This is an SQL WHERE clause - essentially a filter. The browser will update to show only reads matching that filter, and the status line will now show:
- WHERE mapq > 0 AND (flag & 3848) = 0: 76/100 reads
Let’s say our candidate SNV is A->T. We can sort the reads to group reads expressing the variant base at the pileup locus and view them in the alignment pane:
order by base IS 'T' DESC
This is an SQL ORDER BY clause - essentially a sort. DESC is simply the SQL keyword for descending order (ASC is for ascending, which is the default). DESC puts reads that match our condition to the top of the sort for each alignment file in the pileup. The status line will now show:
- WHERE mapq > 0 AND (flag & 3848) = 0: 76/100 reads │ ORDER BY base IS 'T' DESC
The alignment pane now shows only reads matching our WHERE condition, grouped by our ORDER BY condition. We may now explore the pileup visually for any properties of interest, e.g. if most of the reads supporting the variant are soft-clipped.
After exploring the pileup, it can be useful to quantify the locus in more detail. For example, to count the number of reads exhibiting the variant:
count base IS 'T'
This will return a message below the command line:
- base IS 'T' / mapq > 0 AND (flag & 3848) = 0: 24/76 reads
The count returned is the count of reads matching the current WHERE
condition AND the condition provided to count, against the total count
of reads matching the current WHERE condition. If instead you wanted to
count all reads matching a condition and ignoring the current WHERE
condition, use count all [condition]:
count all base IS 'T'
returning:
- base IS 'T': 28/100 reads
From this we may conclude that four reads exhibit the variant, but
are filtered out by our current WHERE clause on mapq and
flag. We may consider relaxing our requirement that the
mate must be mapped to see if this brings any of those four reads into
view. To do so, we can simply grab the current WHERE clause from the
status bar with Ctrl-w. This places it in the command line,
where we may edit and resubmit:
where mapq > 0 AND (flag & 3840 /* edited from 3848 */ ) = 0;
Editing the query is made easy by using the readline-style editing commands detailed in Section 3.1. This concludes the example, but is by no means the limit of what you can do with apb. See the next section for further examples and snippets.
flag is a bitmask of read properties defined by the SAM
format - see https://broadinstitute.github.io/picard/explain-flags.html
from the Broad Institute, which both explains the meaning of the flag
bits and allows you to calcuate the number (like 3848) which corresponds
to a particular set of properties. See https://www.w3schools.com/programming/prog_operators_bitwise.php
for a quick introduction to bitmask syntax. SQLite supports the bitwise
and, or, and not.
In SQL the ORDER BY clause can take multiple keys. This
is supported by the order by command, so we could for
example have written order by base IS 'T' DESC, basequal.
This would have sorted the view such that reads exhibiting a ‘T’ at the
pileup locus will appear first, but now within that group, reads will be
sorted by the quality of that base, in ascending order (lowest
first).
Missing values are encoded as NULL, and different operators behave
differently with NULL values in SQL. For example, a deleted
base is NULL - where base != 'A' will exclude
deletions, as NULL always evaluates to false in comparisons using
!=, i.e. NULL != 'A' is false. Whereas,
where base IS NOT 'A' will include deletions as
NULL IS NOT 'A' evaluates to true. To check for the
presence of missing values for a field, use
where <col> IS NULL.
In SQL, AND conditions always bind more tightly than OR. This means that:
WHERE a AND b AND c OR d
is equivalent to:
WHERE (a AND b AND c) OR d
If you want to bind two conditions together, use brackets:
WHERE a AND b AND (c OR d)
Querying is flexible and powerful. Some useful examples:
When viewing multiple alignment pileups, use instr() to
subset your query to a specific file, e.g.:
where instr(path, 'PD12345a') and base != 'A'
This is made quite simple by the indel column. For
example, to search for an 3-base insertion anchored at the pileup:
where indel = 3
deletions are represented by negative numbers, so for a 3-base deletion:
where indel = -3
Note that these commands search for indels anchored at the pileup base, i.e. beginning immediately after the pileup locus. If you want to search for deleted bases at the pileup locus itself, use:
where is_del
Aux tags may be queried with ->>. For example, to
subset a query to a read group via the RG tag:
where base != 'G' and tags ->> '$.RG' = 'sample1'
For numeric tags, all comparison an arithmetic operators are available:
where tags ->> '$.NM' > 2
A read missing a tag returns null, so:
where tags ->> '$.RG' is null
Would find reads missing the RG tag.
Pattern matching is available on all text fields, such as the CIGAR string. E.g. to filter for reads containing a 3-base insertion at any position:
where instr(cigar, '3I')
or for reads containing soft clipping of any length:
where instr(cigar, 'S')
Arithmetic functions are available. For example:
where end - start < 100
would filter the view to reads less than 100 bases long.
This is a slightly more advanced example. qpos is the
1-based offset into seq for the base at the pileup
position, matching SQLite’s own 1-based substr() indexing,
so to search for the 4-mer GATC starting at the query
position:
where substr(seq, qpos, 4) = 'GATC'
LIKE (_/%) and
GLOB (?/*/[ACG])
both work as wildcards. To search for two possible trinucleotide motifs
at the query position:
where substr(seq, qpos, 3) glob 'A[CG]T'
You can also search for motifs within a window of the
seq string. This command searches for GATC
within the first 10 bases of the read:
where instr(substr(seq, 1, 10), 'GATC') > 0
Any and all of these approaches may be combined, and more is possible. See SQLite’s expression/function reference - https://sqlite.org/lang_expr.html.
dump FunctionalityA dump is a small, self-contained sqlite3 file with just the reads at the specified locus. A few use cases:
sqlite3 my.db allows for more complex analysis if needed
(GROUP BY, aggregates and summaries, etc.).apb if they’re happy just to use
sqlite3).Note that all coordinate data within the dump is 1-based, matching the display.
htslib/samtools/bcftools, and by extension all alignment and VCF data, mix 0-based half-open and 1-based closed coordinate systems. This can be tricky to navigate.
apb uses 1-based closed coordinates throughtout, i.e coordinates
start at 1, and coordinate ranges are [x-y] where both x and y are
included. 1-based closed coordinates match the human-readable VCF and
SAM format specs, and to samtools commands
e.g. samtools view .... Coordinates from these sources,
i.e. a variant locus from a VCF, may be passed directly to the locus CLI
argument at startup. Loci from 0-indexed sources, e.g. BED files, may be
passed directly to the CLI argument if the -0 flag is set,
but the display and internal data will still be in 1-based closed
coordinates. If extracting loci from unspecified or uncertain sources,
make sure to confirm the coordinate system! The 1-based coordinates as
displayed in apb also match the behaviour of both IGV and
jbrowse.
If you are using apb to compare to data as seen in htslib
bam1_t and bam_pileup1_t structs or equivalent
(e.g. pysam alignment records), keep in mind that all coordinate data
will be shifted by 1, including qpos.
The Alt-based command line keybinds (M-b, M-f, M-d and M-Bksp) require the terminal to send Option as Meta. In Terminal, enable “Use Option as Meta key” under Settings > Profiles > Keyboard.
Some command line and navigation keys depend on your keyboard and terminal.
On a mac laptop keyboard, which has no dedicated Home, End or forward Delete keys, use fn+Left and fn+Right for Home and End, and fn+Delete for forward Delete. Ctrl-A and Ctrl-E also move to the start and end of the line. macOS uses Ctrl-Left and Ctrl-Right to switch Spaces, so these may not reach apb. Use Option-Left and Option-Right, or Alt-b and Alt-f, to move by word instead. PgUp and PgDn can be sent using the fn modifier key with the up and down arrow keys if using a mac laptop keyboard.
On Linux and Windows terminals, Alt-based keybinds generally work without extra configuration.
Your experience of apb, and many other terminal-based applications,
may be improved by installing a modern terminal font designed for
legibility. apb recommends 0xProto (https://github.com/0xType/0xProto) try it (and others)
out at https://www.programmingfonts.org/#oxproto.
See the README or the project GitHub (https://github.com/blex-max/apb-pileup-browser) for installation instructions, the project roadmap, and other background.