A hifiasm fork for metagenome assembly using Hifi reads.

Hi, I use hifiasm-meta to assemble urogenital tract metagenomics data from CAMI.

This data was simulated by CAMISIM, average read length: 3,000 bp, read length s.d.: 1,000 bp.

Run log:

$ hifiasm_meta -o cami_0.hifiasm_meta.out -t 32 /database/openstack.cebitec.uni-bielefeld.de/swift/v1/CAMI_Urogenital_tract/pacbio/2018.01.23_14.08.31_sample_0/reads/anonymous_reads.fq.gz

[M::hamt_assemble] Skipped read selection.
[M::ha_analyze_count] lowest: count[16383] = 0
[M::hamt_ft_gen::278.101*[email protected]] ==> filtered out 0 k-mers occurring 750 or more times
[M::hamt_assemble] Generated flt tab.
alloc 1666925 uint16_t
[M::ha_pt_gen::398.464*4.70] ==> counted 131777689 distinct minimizer k-mers
[M::ha_pt_gen] count[16383] = 0 (for sanity check)
[M::ha_analyze_count] lowest: count[16383] = 0
tot_cnt=59765
tot_pos=59765
[M::ha_pt_gen::431.595*5.13] ==> indexed 59765 positions
[M::hamt_assemble::439.470*[email protected]] ==> corrected reads for round 1
[M::hamt_assemble] # bases: 4957619989; # corrected bases: 0; # recorrected bases: 0
[M::hamt_assemble] size of buffer: 0.132GB
[M::ha_pt_gen::470.852*6.04] ==> counted 131777979 distinct minimizer k-mers
[M::ha_pt_gen] count[16383] = 0 (for sanity check)
[M::ha_analyze_count] lowest: count[16383] = 0
tot_cnt=59765
tot_pos=59765
[M::ha_pt_gen::506.590*6.28] ==> indexed 59765 positions
[M::hamt_assemble::514.866*[email protected]] ==> corrected reads for round 2
[M::hamt_assemble] # bases: 4957619989; # corrected bases: 0; # recorrected bases: 0
[M::hamt_assemble] size of buffer: 0.132GB
[M::ha_pt_gen::559.852*6.81] ==> counted 131777979 distinct minimizer k-mers
[M::ha_pt_gen] count[16383] = 0 (for sanity check)
[M::ha_analyze_count] lowest: count[16383] = 0
tot_cnt=59765
tot_pos=59765
[M::ha_pt_gen::597.090*6.98] ==> indexed 59765 positions
[M::hamt_assemble::606.630*[email protected]] ==> corrected reads for round 3
[M::hamt_assemble] # bases: 4957619989; # corrected bases: 0; # recorrected bases: 0
[M::hamt_assemble] size of buffer: 0.132GB
[M::ha_pt_gen::643.258*7.55] ==> counted 131777979 distinct minimizer k-mers
[M::ha_pt_gen] count[16383] = 0 (for sanity check)
[M::ha_analyze_count] lowest: count[16383] = 0
tot_cnt=59765
tot_pos=59765
[M::ha_pt_gen::674.827*7.68] ==> indexed 59765 positions
[M::hamt_assemble::683.525*[email protected]] ==> found overlaps for the final round
[M::ha_print_ovlp_stat] # overlaps: 0
[M::ha_print_ovlp_stat] # strong overlaps: 0
[M::ha_print_ovlp_stat] # weak overlaps: 0
[M::ha_print_ovlp_stat] # exact overlaps: 0
[M::ha_print_ovlp_stat] # inexact overlaps: 0
[M::ha_print_ovlp_stat] # overlaps without large indels: 0
[M::ha_print_ovlp_stat] # reverse overlaps: 0
[M::hist_readlength] <1.0k:
[M::hist_readlength] 1.0k: ]]]]]]]]
[M::hist_readlength] 1.5k: ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
[M::hist_readlength] 2.0k: ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
[M::hist_readlength] 2.5k: ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
[M::hist_readlength] 3.0k: ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
[M::hist_readlength] 3.5k: ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]                                                                                                                                                                                    
[M::hist_readlength] 4.0k: ]]]]]]]]]]]]]]]]]]]]]]]
[M::hist_readlength] 4.5k: ]]]]]]]]]]]]]
[M::hist_readlength] 5.0k: ]]]]]]]
[M::hist_readlength] 5.5k: ]]]]
[M::hist_readlength] 6.0k: ]]
[M::hist_readlength] 6.5k: ]
[M::hist_readlength] 7.0k: ]
[M::hist_readlength] 7.5k: ]
[M::hist_readlength] 8.0k: ]
[M::hist_readlength] 8.5k: ]
[M::hist_readlength] 9.0k: ]
[M::hist_readlength] 9.5k: ]
[M::hist_readlength] 10.0k: ]
[M::hist_readlength] 10.5k: ]
[M::hist_readlength] 11.0k: ]
[M::hist_readlength] 11.5k: ]
[M::hist_readlength] >50.0k: 0
Writing reads to disk...
wrote cmd of length 323: version=0.13-r308, CMD= hifiasm_meta -o cami_0.hifiasm_meta.out -t 32 /database/openstack.cebitec.uni-bielefeld.de/swift/v1/CAMI_Urogenital_tract/pacbio/2018.01.23_14.08.31_sample_0/reads/anonymous_reads.fq.gz
Bin file was created on Wed Dec 30 15:31:02 2020
Hifiasm_meta 0.1-r022 (hifiasm code base 0.13-r308).
Reads has been written.
[hamt::write_All_reads] Writing per-read coverage info...
[hamt::write_All_reads] Finished writing.
Writing ma_hit_ts to disk...
ma_hit_ts has been written.
Writing ma_hit_ts to disk...
ma_hit_ts has been written.
bin files have been written.
Writing raw unitig GFA to disk...
[M::hamt_output_unitig_graph_advance] Writing GFA...
[M::hamt_output_unitig_graph_advance] Writing GFA...
[M::hamt_output_unitig_graph_advance] Writing GFA...
Inconsistency threshold for low-quality regions in BED files: 70%
Writing debug asg to disk...
[M::write_debug_assembly_graph] took 0.02s

[M::main] Hifiasm code base version: 0.13-r308
[M::main] Hifiasm_meta version: 0.1-r022
[M::main] CMD: hifiasm_meta -o cami_0.hifiasm_meta.out -t 32 /database/openstack.cebitec.uni-bielefeld.de/swift/v1/CAMI_Urogenital_tract/pacbio/2018.01.23_14.08.31_sample_0/reads/anonymous_reads.fq.gz
[M::main] Real time: 691.048 sec; CPU: 5463.747 sec; Peak RSS: 16.432 GB

Output:

$ ll
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.a_ctg.gfa
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.a_ctg.noseq.gfa
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.dbg_asg
.rw-r--r-- zhujie 2782  1.2 GB Wed Dec 30 15:31:04 2020 cami_0.hifiasm_meta.out.ec.bin
.rw-r--r-- zhujie 2782 38.2 MB Wed Dec 30 15:31:04 2020 cami_0.hifiasm_meta.out.ec.mt.bin
.rw-r--r-- zhujie 2782  6.7 MB Wed Dec 30 15:31:00 2020 cami_0.hifiasm_meta.out.ovecinfo.bin
.rw-r--r-- zhujie 2782  9.5 MB Wed Dec 30 15:31:04 2020 cami_0.hifiasm_meta.out.ovlp.reverse.bin
.rw-r--r-- zhujie 2782  9.5 MB Wed Dec 30 15:31:04 2020 cami_0.hifiasm_meta.out.ovlp.source.bin
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.p_ctg.gfa
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.p_ctg.noseq.gfa
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.p_utg.gfa
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.p_utg.noseq.gfa
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.r_utg.gfa
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.r_utg.lowQ.bed
.rw-r--r-- zhujie 2782    0 B  Wed Dec 30 15:31:07 2020 cami_0.hifiasm_meta.out.r_utg.noseq.gfa

All GFA file size is zero.

Any help? Thanks ~

Hi, We are struggling to perform de novo assembly of meta bacterial samples selectively cultured with antimicrobials from wasterwater using hifiasm-meta with the default parameters. The sequencing depth seemed to be fine, but the number of circulated bacterial genomes and plasmids is not large, so the resulted contigs would not be good. We guess the cause might be due to the increased redundancy of sequences (bacterial species and plasmids). Someone knows if there are any effective settings to deal with this kind of data? Thanks!

Hello,

I performed de novo assembly on two human faecal metagenomes sequenced with PacBio Sequel II. I tested metaFlye (2.9-b1768) and hifiasm-meta (v0.2.1). As you can see below, hifiasm-meta produces much larger assemblies.

I mapped on the PacBio assemblies Illumina paired-end reads obtained from the same samples. Even if the assemblies of hifiasm_meta are much larger, the proportion of mapped reads only increases slightly. In addition, the proportion of reads aligned exactly 1 time is much lower. This suggests that hifiasm-meta produces redundant assemblies. What do you think?

Thanks for you help, Florian

Donor 1

| | metaFlye | hifiasm_meta | | ---------------------------------------------------- | ----------- | ------------- | | assembly size (bp) | 596 522 308 | 831 187 874 | | # contigs | 9 253 | 15 586 | | N50 (bp) | 164 736 | 132 052 | | % illumina reads aligned concordantly exactly 1 time | 50.79 | 39.45 | | % illumina reads aligned concordantly > 1 time | 23.50 | 38.31 | | % illumina reads aligned concordantly | 74.29 | 77.76 |

Donor 2

| | metaFlye | hifiasm_meta | | ---------------------------------------------------- | ----------- | ------------- | | assembly size (bp) | 264 656 715 | 551 812 461 | | # contigs | 3 836 | 17 080 | | N50 (bp) | 243 801 | 44 732 | | % illumina reads aligned concordantly exactly 1 time | 55.28 | 20.34 | | % illumina reads aligned concordantly > 1 time | 33.15 | 74.26 | | % illumina reads aligned concordantly | 88.43 | 94.6 |

Hi,

I have tested hifiasm-meta on a pacbio hifi data obtained from fecal metagenome of a healthy human.

Below are the library statistics:

sum = 13017330229, n = 1646208, ave = 7907.46, largest = 21324
N50 = 8596, n = 605631
N60 = 7871, n = 763863
N70 = 7149, n = 937306
N80 = 6377, n = 1129752
N90 = 5392, n = 1350332
N100 = 104, n = 1646208

Below are the assembly statistics (asm.p_ctg.gfa):

sum = 831324548, n = 15560, ave = 53427.03, largest = 3704035
N50 = 132051, n = 896
N60 = 73324, n = 1769
N70 = 45743, n = 3226
N80 = 29874, n = 5501
N90 = 19672, n = 8924
N100 = 2682, n = 15560

Unfortunately, it seems that there are no circular contigs even if some contigs are very long (>3Mb) Here is a this screenshot:

Is there something i'm doing wrong ?

Thanks for your help, Florian

This project is quite exciting, but like you mentioned in your pre-print, there is very little public training data to help optimize for this use-case.

I'd like to point the authors to a substantially larger and more representative dataset: 11 real individual human HiFi fecal metagenomes (which are NOT pooled). They have a more realistic distribution of species (some highly abundant but many lower-abundant ones).

PRJNA754443 11_sra_samples.csv

Expected differences seen in this real dataset compared to the "pooled" samples used to benchmark this:

1. These new samples have less equitable (but arguably more realistic) distributions of microbes than the pooled samples because you aren't merging multiple non-overlapping sets of high-abundance bugs; there is more of an exponential decay in abundances.
2. These new samples would be expected to have potentially less tangled graphs, as they are less likely to contain mixtures of near-identical strains from different people in the same sample. Large numbers of closely-related genomes are less likely to be found within a given individual when evolutionary selection has taken place to limit the diversity of closely-related strains competing for the same resources/niches within the gut
3. Overall depth is slightly lower with a median of roughly 1 million reads of 7kb length.
4. Despite point 3, there may be more potential to capture rare microbes because these single samples have twice the effective read depth per human subject than the pooled samples which ostensibly have twice the volume of data in total.

I've run the latest version of this assembler on these samples already, and see substantially fewer closed genomes (and overall HQ mags!) per sample than the pooled samples, as expected. I aim to do numerous more experiments with some of the recent cleaning options and potentially other (graph-aware?) binning tweaks, but I don't expect the overall picture to change much.

I'm curious to see whether further improvements can be made given the availability of this larger corpus of individual-level human microbiome HiFi data.

The manuscript briefly mentions how Hifiasm-meta uses a new method for filtering contained reads. I'm interested in learning about the filtering mechanism here. Could you please share more details of the algorithm ; OR point me to appropriate place in the code. Pasting the text from your manuscript:

Treatment of contained reads. The standard procedure to construct a string graph discards a read contained in a longer read. This may lead to an assembly gap if the contained read and the longer read actually reside on different haplotypes10. The original hifiasm patches such gaps by rescuing contained reads after graph construction. Hifiasm-meta tries to resolve the issue before graph construction instead. It retains a contained read if other reads exactly overlapping with the read are inferred to come from different haplotypes. In other words, hifiasm-meta only drops a contained read if there are no other similar haplotypes around it. This strategy often retains extra contained reads that are actually redundant. These extra reads usually lead to bubble-like subgraphs and are later removed by the bubble popping algorithm in the original hifiasm.

I wish to understand the exact condition / threshold values which decides whether to retain the contained read.

Thank you.

Hi xfengenfx

recently，i use the hifiam-meta to assemble my metagenomic HIFI data，i encountered same error in two times at two compute cluster，which shows stop at the Writing GFA step suddenly. here is my two log file, the first one was in the slurm system,the second one was in the usual system. so i can't get my final contig GFA file ,can you figure it out for me. job-26237_1.err.txt nohup.out.txt appreciate it

Hi Xiaowen, I am wondering if it is possible to obtain a list of reads that contribute to each contig in the assembly?

This seems like it would be highly valuable for metagenomics, as it can identify all reads associated with specific bacterial genomes. In addition, it would be extremely valuable for a more specific use-case I describe below.

I am working on a problem where I am trying to assemble an endosymbiont bacteria from a larger HiFi dataset focused on the host organism. The assembly of the full dataset with hifiasm did not produce a complete bacteria contig, it was present as several smaller contigs. I attempted to re-assemble and improve the quality of these results. To accomplish this, I have:

1. Mapped contigs from a hifiasm assembly of the full dataset to a reference of the target bacteria, to identify and extract relevant bacteria contigs.
2. Mapped reads to those putative bacteria contigs to identify reads that are most likely target bacteria, and extract them.
3. Performed assembly with this subset of putative bacteria reads using hifiasm-meta.

This resulted in a complete, circular genome for the target bacteria, along with a few small tangled contigs, suggesting the approach worked pretty well. The small contigs in the new assembly are likely some combination of host reads and perhaps strain variation.

The genome has a few frameshifts and I would like to try polishing it using only the reads that were used to build the complete bacteria contig. I have used minimap2 to align the subset of reads to this contig, and there are several short regions in which some proportion of reads map poorly (alignments are <1000 bp and they are being hard clipped >3000 bp on each side). I think these are potentially host reads. I can filter these out using samclip, but it would be helpful to know whether or not they were used to construct this contig, and therefore deserve to be excluded.

Given metagenomic assemblies often result in several complete genomes, I think the same topic will come up. Polishing would also be desirable, but problematic read alignments would be more prevalent due to more species, shared repeats, etc. Having the ability to assign reads to particular contigs would be a tremendous help here too.

Any advice would be greatly appreciated!

Thanks, Dan

Hi, are you back-porting (up-porting? side-porting?) the Hi-C integration from hifiasm? We are sequencing some species where up to half the sample might be bacteria and fungi (the target species is a plant), and are considering using hifiasm-meta for this as the first step, and then mapping and extracting the plant-specific reads for a separate assembly with regular hifiasm. We are also getting Hi-C reads for these samples, so I wondered if Hi-C integration might be helpful for separating species in hifiasm-meta.

Sincerely, Ole

Hello,

Could you please explain more about the S-line of the noseq.gfa file. What does "dp" and "ts" represent for, respectively.

thank you.

Hello, xfengnefx!

With NGS shotgun reads, to get MAGs we usually assemble pair-end reads into contigs, and then recover MAGs through binning.

What I want to ask is that for HiFi reads, in order to get MAGs with higher quality whether it is necessary to conduct binning after we get contigs using hifiasm-meta?

Thanks for your helping.

Hello

I am a beginner and I have a question about metagenomic analysis using HiFi PacBio long reads. In short read metagenomics I have seen in some papers who suggest doing taxonomic and functional profiling after assembly, to increase the precision. I was wondering if with long reads we can directly use the raw reads for profiling or it is still better to perform assembly first.

Thank you

hello

i test assembly efficiency of hifiasm-meta and metaflye with mock communty (MSA 1003).

For f5bcb58692924cb7_1 (ATCC-12228 , len: 2503245 bp), hifiasm-meta got 544 contigs, the longest one is 2387482 bp, and the others are shorter than 30000 bp. when I mapped these contigs to the reference genome, I found high redundancy among these contigs, especially the longest contig included lots of shorter contigs. On the other hand, metaflye got one contig, and exactly the length of the reference genome. But for 5964adb8d0df4fde_1 (ATCC-33323, len: 1854273), hifiasm-meta got 8 contigs, the coverage is good and almost no overlap existed among these 8 contigs.

So, i want to ask : 1, why different assembly results appeared for different reference genome; 2, how should I set parameters to get a set of contigs with low redundancy while maintaining high coverage.

the current parameters i set was: hifiasm_meta -t 36 --force-rs -o mock2 ../mock2.fastq.gz

thanks for your help

Hello

It seems there are duplicate edges in the produced GFA. What is the purpose of these?

E.g. if we'd take sheepB.hifiasm-meta.a_ctg.gfa.gz then we'll end with:

L       s0.ctg000590l   +       s0.ctg027907l   -       10632M  L1:i:29150
...
L       s0.ctg027907l   +       s0.ctg000590l   -       10637M  L1:i:14435

Note that overlaps are different as well which does look suspicious...