An Efficient Parallel Coding Scheme in Erasure-Coded Storage Systems
Erasure codes have been considered as one of the most promising techniques for data reliability enhancement and storage efficiency in modern distributed storage systems. However, erasure codes often suffer from a time-consuming coding process which makes them nearly impractical. The opportunity to s...
Saved in:
| Published in | IEICE Transactions on Information and Systems Vol. E101.D; no. 3; pp. 627 - 643 |
|---|---|
| Main Authors | , |
| Format | Journal Article |
| Language | English |
| Published |
Tokyo
The Institute of Electronics, Information and Communication Engineers
01.03.2018
Japan Science and Technology Agency |
| Subjects | |
| Online Access | Get full text |
| ISSN | 0916-8532 1745-1361 1745-1361 |
| DOI | 10.1587/transinf.2017EDP7218 |
Cover
| Abstract | Erasure codes have been considered as one of the most promising techniques for data reliability enhancement and storage efficiency in modern distributed storage systems. However, erasure codes often suffer from a time-consuming coding process which makes them nearly impractical. The opportunity to solve this problem probably rely on the parallelization of erasure-code-based application on the modern multi-/many-core processors to fully take advantage of the adequate hardware resources on those platforms. However, the complicated data allocation and limited I/O throughput pose a great challenge on the parallelization. To address this challenge, we propose a general multi-threaded parallel coding approach in this work. The approach consists of a general multi-threaded parallel coding model named as MTPerasure, and two detailed parallel coding algorithms, named as sdaParallel and ddaParallel, respectively, adapting to different I/O circumstances. MTPerasure is a general parallel coding model focusing on the high level data allocation, and it is applicable for all erasure codes and can be implemented without any modifications of the low level coding algorithms. The sdaParallel divides the data into several parts and the data parts are allocated to different threads statically in order to eliminate synchronization latency among multiple threads, which improves the parallel coding performance under the dummy I/O mode. The ddaParallel employs two threads to execute the I/O reading and writing on the basis of small pieces independently, which increases the I/O throughput. Furthermore, the data pieces are assigned to the coding thread dynamically. A special thread scheduling algorithm is also proposed to reduce thread migration latency. To evaluate our proposal, we parallelize the popular open source library jerasure based on our approach. And a detailed performance comparison with the original sequential coding program indicates that the proposed parallel approach outperforms the original sequential program by an extraordinary speedups from 1.4x up to 7x, and achieves better utilization of the computation and I/O resources. |
|---|---|
| AbstractList | Erasure codes have been considered as one of the most promising techniques for data reliability enhancement and storage efficiency in modern distributed storage systems. However, erasure codes often suffer from a time-consuming coding process which makes them nearly impractical. The opportunity to solve this problem probably rely on the parallelization of erasure-code-based application on the modern multi-/many-core processors to fully take advantage of the adequate hardware resources on those platforms. However, the complicated data allocation and limited I/O throughput pose a great challenge on the parallelization. To address this challenge, we propose a general multi-threaded parallel coding approach in this work. The approach consists of a general multi-threaded parallel coding model named as MTPerasure, and two detailed parallel coding algorithms, named as sdaParallel and ddaParallel, respectively, adapting to different I/O circumstances. MTPerasure is a general parallel coding model focusing on the high level data allocation, and it is applicable for all erasure codes and can be implemented without any modifications of the low level coding algorithms. The sdaParallel divides the data into several parts and the data parts are allocated to different threads statically in order to eliminate synchronization latency among multiple threads, which improves the parallel coding performance under the dummy I/O mode. The ddaParallel employs two threads to execute the I/O reading and writing on the basis of small pieces independently, which increases the I/O throughput. Furthermore, the data pieces are assigned to the coding thread dynamically. A special thread scheduling algorithm is also proposed to reduce thread migration latency. To evaluate our proposal, we parallelize the popular open source library jerasure based on our approach. And a detailed performance comparison with the original sequential coding program indicates that the proposed parallel approach outperforms the original sequential program by an extraordinary speedups from 1.4x up to 7x, and achieves better utilization of the computation and I/O resources. |
| Author | LIU, Guangming DONG, Wenrui |
| Author_xml | – sequence: 1 fullname: LIU, Guangming organization: National Supercomputer Center in Tianjin – sequence: 1 fullname: DONG, Wenrui organization: National University of Defense Technology |
| BookMark | eNqFkM1qGzEURkVJoHaSN-hioOtx9S9Nd8F2m0AgIU7WQtbcSWTGGleSCX77yjh2SxfNSgKd8-neb4zOwhAAoS8ET4jQ6luONiQfugnFRM1nD4oS_QmNiOKiJkySMzTCDZG1Fox-RuOUVhgTTYkYodl1qOZd552HkKsHG23fQ19Nh9aHl2rhXmENlS9MtGkboS4P0FaLPET7AtVilzKs0yU672yf4Or9vEDPP-ZP05v67v7n7fT6rnZcylxTvFx2LZcUNNMNuLbhmDe0kU46x9VSg9JEW8s4l6ppMSvrUNF0QiqmWavYBRKH3G3Y2N1bGdVsol_buDMEm30V5liF2VcB7WZfRfG-HrxNHH5tIWWzGrYxlFH3GBdScMoLxQ-Ui0NKEbr_h7_3XLTv_2jOZ5v9EAru-4_kx4O8SrkUevrRxuxdD3-kOcHEzAw7Xv4KOcHu1UYDgf0G8ECjLw |
| CitedBy_id | crossref_primary_10_1007_s11277_020_07937_4 crossref_primary_10_1111_mice_12737 |
| Cites_doi | 10.1145/258533.258573 10.1109/12.364531 10.1145/258533.258660 10.1109/CC.2013.6506936 10.1109/TIT.2006.874390 10.1109/TIT.2010.2054295 10.1007/978-3-642-28145-7_35 10.1109/18.746809 10.1109/DSN.2005.21 10.1109/18.910576 10.1109/IWQoS.2009.5201391 10.1147/rd.111.0025 10.1109/TC.2007.70830 10.1016/j.future.2011.08.005 10.1007/3-540-49543-6_14 10.1109/NCA.2008.45 10.1109/SNAPI.2010.16 10.1007/11558989_21 10.1016/j.camwa.2006.05.002 10.1109/IPDPS.2008.4536322 10.1109/PDSW.2008.4811887 10.1145/1165389.945450 10.1007/s10115-013-0638-6 10.1145/301250.301441 10.1109/DSN.2006.40 10.3724/SP.J.1001.2012.04175 10.1145/1713072.1713075 10.1109/ISIT.2004.1365074 10.1007/3-540-45748-8_31 10.1109/TIT.2011.2159049 |
| ContentType | Journal Article |
| Copyright | 2018 The Institute of Electronics, Information and Communication Engineers Copyright Japan Science and Technology Agency 2018 |
| Copyright_xml | – notice: 2018 The Institute of Electronics, Information and Communication Engineers – notice: Copyright Japan Science and Technology Agency 2018 |
| DBID | AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D ADTOC UNPAY |
| DOI | 10.1587/transinf.2017EDP7218 |
| DatabaseName | CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional Unpaywall for CDI: Periodical Content Unpaywall |
| DatabaseTitle | CrossRef Computer and Information Systems Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Advanced Technologies Database with Aerospace ProQuest Computer Science Collection Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Computer and Information Systems Abstracts |
| Database_xml | – sequence: 1 dbid: UNPAY name: Unpaywall url: https://proxy.k.utb.cz/login?url=https://unpaywall.org/ sourceTypes: Open Access Repository |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Computer Science |
| EISSN | 1745-1361 |
| EndPage | 643 |
| ExternalDocumentID | 10.1587/transinf.2017edp7218 10_1587_transinf_2017EDP7218 article_transinf_E101_D_3_E101_D_2017EDP7218_article_char_en |
| GroupedDBID | -~X 5GY ABJNI ABZEH ACGFS ADNWM AENEX ALMA_UNASSIGNED_HOLDINGS CS3 DU5 EBS EJD F5P ICE JSF JSH KQ8 OK1 P2P RJT RZJ TN5 ZKX 1TH AAYXX AFFNX C1A CITATION CKLRP H13 RYL VOH ZE2 ZY4 7SC 8FD JQ2 L7M L~C L~D ADTOC UNPAY |
| ID | FETCH-LOGICAL-c466t-20bbfd462e8389ecd94049296c6cc47b8e7818aa344679d03201259f567383d73 |
| IEDL.DBID | UNPAY |
| ISSN | 0916-8532 1745-1361 |
| IngestDate | Tue Aug 19 18:26:15 EDT 2025 Mon Jun 30 12:13:35 EDT 2025 Thu Apr 24 22:53:27 EDT 2025 Wed Oct 01 05:56:37 EDT 2025 Wed Sep 03 06:22:43 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 3 |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c466t-20bbfd462e8389ecd94049296c6cc47b8e7818aa344679d03201259f567383d73 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| OpenAccessLink | https://proxy.k.utb.cz/login?url=https://www.jstage.jst.go.jp/article/transinf/E101.D/3/E101.D_2017EDP7218/_pdf |
| PQID | 2014565424 |
| PQPubID | 2048497 |
| PageCount | 17 |
| ParticipantIDs | unpaywall_primary_10_1587_transinf_2017edp7218 proquest_journals_2014565424 crossref_primary_10_1587_transinf_2017EDP7218 crossref_citationtrail_10_1587_transinf_2017EDP7218 jstage_primary_article_transinf_E101_D_3_E101_D_2017EDP7218_article_char_en |
| ProviderPackageCode | CITATION AAYXX |
| PublicationCentury | 2000 |
| PublicationDate | 2018-03-01 |
| PublicationDateYYYYMMDD | 2018-03-01 |
| PublicationDate_xml | – month: 03 year: 2018 text: 2018-03-01 day: 01 |
| PublicationDecade | 2010 |
| PublicationPlace | Tokyo |
| PublicationPlace_xml | – name: Tokyo |
| PublicationTitle | IEICE Transactions on Information and Systems |
| PublicationTitleAlternate | IEICE Trans. Inf. & Syst. |
| PublicationYear | 2018 |
| Publisher | The Institute of Electronics, Information and Communication Engineers Japan Science and Technology Agency |
| Publisher_xml | – name: The Institute of Electronics, Information and Communication Engineers – name: Japan Science and Technology Agency |
| References | [38] IEEE Computer Society, The Hadoop Distributed File System, Lake Tahoe, Nevada, USA, 2010. [43] D. Borthakur, R. Schmit, R. Vadali, S. Chen, and P. Kling, “Hdfs raid,” Tech talk, Yahoo Developer Network, 2010. [11] R.M. Tomasulo, “An efficient algorithm for exploiting multiple arithmetic units,” IBM J. Res. & Dev., vol.11, no.1, pp.25-33, 1967. [26] L. Xu and J. Bruck, “X-code: MDS array codes with optimal encoding,” IEEE Trans. Inf. Theory, vol.45, no.1, pp.272-276, 1999. 10.1109/18.746809 [10] B. Fan, W. Tantisiriroj, L. Xiao, and G. Gibson, “Diskreduce: Replication as a prelude to erasure coding in data-intensive scalable computing,” the International Conference for High Performance Computing, Networking, Storage and Analysis (SC11), Nov. 12-18, 2011. [25] M. Blaum, J. Brady, J. Bruck, and J. Menon, “Evenodd: An efficient scheme for tolerating double disk failures in raid architectures,” IEEE Trans. Comput., vol.44, no.2, pp.192-202, 1995. 10.1109/12.364531 [5] J.S. Plank and L. Xu, “Optimizing cauchy reed-solomon codes for fault-tolerant network storage applications,” Fifth IEEE International Symposium on Network Computing and Applications, NCA 2006, 24-26 July 2006, Cambridge, Massachusetts, USA, pp.173-180, 2006. 10.1109/nca.2006.43 [35] R. Wyrzykowski, L. Kuczynski, and M. Wozniak, “Towards efficient execution of erasure codes on multicore architectures,” Proc. 10th international conference on Applied Parallel and Scientific Computing-Volume 2, PARA'10, Berlin, Heidelberg, vol.7134, pp.357-367, Springer-Verlag, 2012. 10.1007/978-3-642-28145-7_35 [4] J. Blömer and J.-P. Seifert, “On the complexity of computing short linearly independent vectors and short bases in a lattice,” Proc. Thirty-First Annual ACM Symposium on Theory of Computing, May 1-4, 1999, Atlanta, Georgia, USA, pp.711-720, 1999. 10.1145/301250.301441 [41] D. Ford, F. Labelle, F.I. Popovici, M. Stokely, V.A. Truong, L. Barroso, C. Grimes, and S. Quinlan, “Availability in globally distributed storage systems,” Proc. 9th USENIX conference on Operating systems design and implementation, pp.1-7, USENIX Association, 2010. [40] M. Luby, “Tornado Codes: Practical erasure codes based on random irregular graphs,” International Workshop on Randomization and Approximation Techniques in Computer Science (RANDOM '98), Barcelona, Spain, vol.1518, pp.171-175, USENIX Association, 1998. 10.1007/3-540-49543-6_14 [3] D. Borthakur, “The hadoop distributed file system: Architecture and design,” Hadoop Project Website, vol.11, p.21, 2007. [33] M.L. Curry, A. Skjellum, H.L. Ward, and R. Brightwell, “Accelerating reed-solomon coding in raid systems with gpus,” IEEE International Symposium on Parallel and Distributed Processing (IPDPS 2008), pp.1-6, 2008. 10.1109/ipdps.2008.4536322 [18] M.G. Luby, M. Mitzenmacher, M.A. Shokrollahi, D.A. Spielman, and V. Stemann, “Practical loss-resilient codes,” 29th Annual ACM Symposium on Theory of Computing, El Paso, TX, USA, pp.150-159, ACM, 1997. 10.1145/258533.258573 [29] K.V. Rashmi, N.B. Shah, and P.V. Kumar, “Optimal exact-regenerating codes for distributed storage at the MSR and MBR points via a product-matrix construction,” IEEE Trans. Inf. Theory, vol.57, no.8, pp.5227-5239, 2011. 10.1109/tit.2011.2159049 [21] J.L. Hafner, “Weaver codes highly fault tolerant erasure codes for storage systems,” Proc. 4th conference on USENIX Conference on File and Storage Technologies, San Francisco, USA, p.16, Usenix Association, 2005. 10.1109/acssc.2011.6190329 [9] B. Fan, W. Tantisiriroj, L. Xiao, and G. Gibson, “Diskreduce: Raid for data-intensive scalable computing,” Proc. Sumpercomping-Petascale Data Storage Workshop (PDSW'09), Portland, pp.6-10, ACM, 2009. 10.1145/1713072.1713075 [27] C. Huang and L. Xu, “Star: An efficient coding scheme for correcting triple storage node failures,” IEEE Trans. Comput., vol.57, no.7, pp.889-901, 2008. 10.1109/tc.2007.70830 [31] J. Li, S. Yang, X. Wang, X. Xue, and B. Li, “Tree-structured data regeneration with network coding in distributed storage systems,” Proc. 17th International Workshop on Quality of Service (IWQoS), 2009, pp.1-9, 2009. 10.1109/iwqos.2009.5201391 [2] S. Ghemawat, H. Gobioff, and S.-T. Leung, “The google file system,” Proceedings of ACM SIGOPS Operating Systems Review, vol.37, no.5, pp.29-43, ACM, 2003. 10.1145/1165389.945450 [13] H. Weatherspoon and J. Kubiatowicz, “Erasure coding vs. replication: A quantitative comparison,” Peer-to-Peer Systems, vol.2429, pp.328-337, 2002. 10.1007/3-540-45748-8_31 [14] R. Rodrigues and B. Liskov, “High availability in dhts: Erasure coding vs. replication,” Peer-to-Peer Systems IV, vol.3640, pp.226-239, 2005. 10.1007/11558989_21 [17] S.B. Wicker, Reed-Solomon Codes and Their Applications, pp.1-15, IEEE Press Piscataway, 1994. [1] Y.-J. Wang, W.-D. Sun, S. Zhou, X.-Q. Pei, and X.-Y. Li, “Key technologies of distributed storage for cloud computing,” Journal of Software, vol.23, no.4, pp.962-986, 2012. 10.3724/sp.j.1001.2012.04175 [37] P. Sobe, “Parallel reed/solomon coding on multicore processors,” 2010 International Workshop on Storage Network Architecture and Parallel I/Os (SNAPI), pp.71-80, 2010. 10.1109/snapi.2010.16 [22] M.G. Luby, M. Mitzenmacher, M.A. Shokrollahi, and D.A. Spielman, “Improved low-density parity-check codes using irregular graphs,” IEEE Trans. Inf. Theory, vol.47, no.2, pp.585-598, 2001. 10.1109/18.910576 [39] Y. Wang and S. Li, “Research and performance evaluation of data replication technology in distributed storage systems,” Comput. Math. Appl., vol.51, no.11, pp.1625-1632, 2006. 10.1016/j.camwa.2006.05.002 [6] F.J. MacWilliams and N.J.A. Sloane, The Theory of Error-Correcting Codes, North-Holland, 1977. [36] P. Sobe and K. Peter, “Flexible parameterization of xor based codes for distributed storage,” 2008 7th IEEE International Symposium on Network Computing and Applications, pp.101-110, 2008. 10.1109/nca.2008.45 [16] Y. Wang, X. Li, X. Li, and Y. Wang, “A survey of queries over uncertain data,” Knowledge and Information Systems, vol.37, no.3, pp.485-530, 2013. 10.1007/s10115-013-0638-6 [20] J.L. Hafner, “Hover erasure codes for disk arrays,” 2006 International Conference on Dependable Systems and Networks (DSN 2006), pp.217-226, 2006. 10.1109/dsn.2006.40 [34] M.L. Curry, A. Skjellum, H.L. Ward, and R. Brightwell, “Arbitrary dimension reed-solomon coding and decoding for extended raid on GPUs,” 2008 3rd Petascale Data Storage Workshop, pp.1-3, 2008. 10.1109/pdsw.2008.4811887 [24] A. Shokrollahi, “Raptor codes,” IEEE Trans. Inf. Theory, vol.52, no.6, pp.2551-2567, 2006. 10.1109/tit.2006.874390 [12] D. Karger, E. Lehman, T. Leighton, R. Panigrahy, M. Levine, and D. Lewin, “Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web,” Proc. Twenty-ninth Annual ACM Symposium on Theory of Computing, pp.654-663, ACM, 1997. 10.1145/258533.258660 [7] J.S. Plank, J. Luo, C.D. Schuman, L. Xu, and Z. Wilcox-O'Hearn, “A performance evaluation and examination of open-source erasure coding libraries for storage,” Proc. 7th USENIX Conference on File and Storage Technologies, pp.253-265, USENIX Association, 2009. [8] HDFS-RAID, https://wiki.apache.org/hadoop/HDFS-RAID [32] W. Sun, Y. Wang, and X. Pei, “Tree-structured parallel regeneration for multiple data losses in distributed storage systems based on erasure codes,” China Communications, vol.10, no.4, pp.113-125, 2013. 10.1109/cc.2013.6506936 [19] R.L. Collins and J.S. Plank, “Assessing the performance of erasure codes in the wide-area,” International Conference on Dependable Systems and Networks, Yokohama, Japan, IEEE, pp.182-187, 2005. 10.1109/dsn.2005.21 [28] P. Corbett, B. English, A. Goel, T. Grcanac, S. Kleiman, J. Leong, and S. Sankar, “Row-diagonal parity for double disk failure correction,” Proc. 3rd USENIX Conference on File and Storage Technologies, pp.1-14, 2004. [15] X. Lu, H. Wang, J. Wanga, J. Xu, and D. Li, “Internet-based virtual computing environment: Beyond the data center as a computer,” Future Generation Computer Systems, vol.29, no.1, pp.309-322, 2013. 10.1016/j.future.2011.08.005 [23] R. Karp, M. Luby, and A. Shokrollahi, “Finite length analysis of LT codes,” International Symposium onInformation Theory, 2004, Proceedings., p.37, 2004. 10.1109/isit.2004.1365074 [30] A.G. Dimakis, P.B. Godfrey, Y. Wu, M.J. Wainwright, and K. Ramchandran, “Network coding for distributed storage systems,” IEEE Trans. Inf. Theory, vol.56, no.9, pp.4539-4551, 2010. 10.1109/tit.2010.2054295 [42] C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li, and S. Yekhanin, “Erasure coding in windows azure storage,” USENIX Annual Technical Conference (USENIX ATC), 2012. 22 23 24 25 26 27 28 29 30 31 10 32 11 33 12 34 13 35 14 36 15 37 16 38 17 39 18 19 1 2 3 4 5 6 7 8 9 40 41 20 42 21 43 |
| References_xml | – reference: [28] P. Corbett, B. English, A. Goel, T. Grcanac, S. Kleiman, J. Leong, and S. Sankar, “Row-diagonal parity for double disk failure correction,” Proc. 3rd USENIX Conference on File and Storage Technologies, pp.1-14, 2004. – reference: [29] K.V. Rashmi, N.B. Shah, and P.V. Kumar, “Optimal exact-regenerating codes for distributed storage at the MSR and MBR points via a product-matrix construction,” IEEE Trans. Inf. Theory, vol.57, no.8, pp.5227-5239, 2011. 10.1109/tit.2011.2159049 – reference: [35] R. Wyrzykowski, L. Kuczynski, and M. Wozniak, “Towards efficient execution of erasure codes on multicore architectures,” Proc. 10th international conference on Applied Parallel and Scientific Computing-Volume 2, PARA'10, Berlin, Heidelberg, vol.7134, pp.357-367, Springer-Verlag, 2012. 10.1007/978-3-642-28145-7_35 – reference: [15] X. Lu, H. Wang, J. Wanga, J. Xu, and D. Li, “Internet-based virtual computing environment: Beyond the data center as a computer,” Future Generation Computer Systems, vol.29, no.1, pp.309-322, 2013. 10.1016/j.future.2011.08.005 – reference: [20] J.L. Hafner, “Hover erasure codes for disk arrays,” 2006 International Conference on Dependable Systems and Networks (DSN 2006), pp.217-226, 2006. 10.1109/dsn.2006.40 – reference: [24] A. Shokrollahi, “Raptor codes,” IEEE Trans. Inf. Theory, vol.52, no.6, pp.2551-2567, 2006. 10.1109/tit.2006.874390 – reference: [42] C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li, and S. Yekhanin, “Erasure coding in windows azure storage,” USENIX Annual Technical Conference (USENIX ATC), 2012. – reference: [32] W. Sun, Y. Wang, and X. Pei, “Tree-structured parallel regeneration for multiple data losses in distributed storage systems based on erasure codes,” China Communications, vol.10, no.4, pp.113-125, 2013. 10.1109/cc.2013.6506936 – reference: [40] M. Luby, “Tornado Codes: Practical erasure codes based on random irregular graphs,” International Workshop on Randomization and Approximation Techniques in Computer Science (RANDOM '98), Barcelona, Spain, vol.1518, pp.171-175, USENIX Association, 1998. 10.1007/3-540-49543-6_14 – reference: [34] M.L. Curry, A. Skjellum, H.L. Ward, and R. Brightwell, “Arbitrary dimension reed-solomon coding and decoding for extended raid on GPUs,” 2008 3rd Petascale Data Storage Workshop, pp.1-3, 2008. 10.1109/pdsw.2008.4811887 – reference: [23] R. Karp, M. Luby, and A. Shokrollahi, “Finite length analysis of LT codes,” International Symposium onInformation Theory, 2004, Proceedings., p.37, 2004. 10.1109/isit.2004.1365074 – reference: [18] M.G. Luby, M. Mitzenmacher, M.A. Shokrollahi, D.A. Spielman, and V. Stemann, “Practical loss-resilient codes,” 29th Annual ACM Symposium on Theory of Computing, El Paso, TX, USA, pp.150-159, ACM, 1997. 10.1145/258533.258573 – reference: [43] D. Borthakur, R. Schmit, R. Vadali, S. Chen, and P. Kling, “Hdfs raid,” Tech talk, Yahoo Developer Network, 2010. – reference: [38] IEEE Computer Society, The Hadoop Distributed File System, Lake Tahoe, Nevada, USA, 2010. – reference: [16] Y. Wang, X. Li, X. Li, and Y. Wang, “A survey of queries over uncertain data,” Knowledge and Information Systems, vol.37, no.3, pp.485-530, 2013. 10.1007/s10115-013-0638-6 – reference: [9] B. Fan, W. Tantisiriroj, L. Xiao, and G. Gibson, “Diskreduce: Raid for data-intensive scalable computing,” Proc. Sumpercomping-Petascale Data Storage Workshop (PDSW'09), Portland, pp.6-10, ACM, 2009. 10.1145/1713072.1713075 – reference: [39] Y. Wang and S. Li, “Research and performance evaluation of data replication technology in distributed storage systems,” Comput. Math. Appl., vol.51, no.11, pp.1625-1632, 2006. 10.1016/j.camwa.2006.05.002 – reference: [13] H. Weatherspoon and J. Kubiatowicz, “Erasure coding vs. replication: A quantitative comparison,” Peer-to-Peer Systems, vol.2429, pp.328-337, 2002. 10.1007/3-540-45748-8_31 – reference: [2] S. Ghemawat, H. Gobioff, and S.-T. Leung, “The google file system,” Proceedings of ACM SIGOPS Operating Systems Review, vol.37, no.5, pp.29-43, ACM, 2003. 10.1145/1165389.945450 – reference: [26] L. Xu and J. Bruck, “X-code: MDS array codes with optimal encoding,” IEEE Trans. Inf. Theory, vol.45, no.1, pp.272-276, 1999. 10.1109/18.746809 – reference: [4] J. Blömer and J.-P. Seifert, “On the complexity of computing short linearly independent vectors and short bases in a lattice,” Proc. Thirty-First Annual ACM Symposium on Theory of Computing, May 1-4, 1999, Atlanta, Georgia, USA, pp.711-720, 1999. 10.1145/301250.301441 – reference: [7] J.S. Plank, J. Luo, C.D. Schuman, L. Xu, and Z. Wilcox-O'Hearn, “A performance evaluation and examination of open-source erasure coding libraries for storage,” Proc. 7th USENIX Conference on File and Storage Technologies, pp.253-265, USENIX Association, 2009. – reference: [12] D. Karger, E. Lehman, T. Leighton, R. Panigrahy, M. Levine, and D. Lewin, “Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web,” Proc. Twenty-ninth Annual ACM Symposium on Theory of Computing, pp.654-663, ACM, 1997. 10.1145/258533.258660 – reference: [11] R.M. Tomasulo, “An efficient algorithm for exploiting multiple arithmetic units,” IBM J. Res. & Dev., vol.11, no.1, pp.25-33, 1967. – reference: [33] M.L. Curry, A. Skjellum, H.L. Ward, and R. Brightwell, “Accelerating reed-solomon coding in raid systems with gpus,” IEEE International Symposium on Parallel and Distributed Processing (IPDPS 2008), pp.1-6, 2008. 10.1109/ipdps.2008.4536322 – reference: [27] C. Huang and L. Xu, “Star: An efficient coding scheme for correcting triple storage node failures,” IEEE Trans. Comput., vol.57, no.7, pp.889-901, 2008. 10.1109/tc.2007.70830 – reference: [5] J.S. Plank and L. Xu, “Optimizing cauchy reed-solomon codes for fault-tolerant network storage applications,” Fifth IEEE International Symposium on Network Computing and Applications, NCA 2006, 24-26 July 2006, Cambridge, Massachusetts, USA, pp.173-180, 2006. 10.1109/nca.2006.43 – reference: [21] J.L. Hafner, “Weaver codes highly fault tolerant erasure codes for storage systems,” Proc. 4th conference on USENIX Conference on File and Storage Technologies, San Francisco, USA, p.16, Usenix Association, 2005. 10.1109/acssc.2011.6190329 – reference: [30] A.G. Dimakis, P.B. Godfrey, Y. Wu, M.J. Wainwright, and K. Ramchandran, “Network coding for distributed storage systems,” IEEE Trans. Inf. Theory, vol.56, no.9, pp.4539-4551, 2010. 10.1109/tit.2010.2054295 – reference: [17] S.B. Wicker, Reed-Solomon Codes and Their Applications, pp.1-15, IEEE Press Piscataway, 1994. – reference: [37] P. Sobe, “Parallel reed/solomon coding on multicore processors,” 2010 International Workshop on Storage Network Architecture and Parallel I/Os (SNAPI), pp.71-80, 2010. 10.1109/snapi.2010.16 – reference: [36] P. Sobe and K. Peter, “Flexible parameterization of xor based codes for distributed storage,” 2008 7th IEEE International Symposium on Network Computing and Applications, pp.101-110, 2008. 10.1109/nca.2008.45 – reference: [1] Y.-J. Wang, W.-D. Sun, S. Zhou, X.-Q. Pei, and X.-Y. Li, “Key technologies of distributed storage for cloud computing,” Journal of Software, vol.23, no.4, pp.962-986, 2012. 10.3724/sp.j.1001.2012.04175 – reference: [8] HDFS-RAID, https://wiki.apache.org/hadoop/HDFS-RAID – reference: [41] D. Ford, F. Labelle, F.I. Popovici, M. Stokely, V.A. Truong, L. Barroso, C. Grimes, and S. Quinlan, “Availability in globally distributed storage systems,” Proc. 9th USENIX conference on Operating systems design and implementation, pp.1-7, USENIX Association, 2010. – reference: [10] B. Fan, W. Tantisiriroj, L. Xiao, and G. Gibson, “Diskreduce: Replication as a prelude to erasure coding in data-intensive scalable computing,” the International Conference for High Performance Computing, Networking, Storage and Analysis (SC11), Nov. 12-18, 2011. – reference: [14] R. Rodrigues and B. Liskov, “High availability in dhts: Erasure coding vs. replication,” Peer-to-Peer Systems IV, vol.3640, pp.226-239, 2005. 10.1007/11558989_21 – reference: [6] F.J. MacWilliams and N.J.A. Sloane, The Theory of Error-Correcting Codes, North-Holland, 1977. – reference: [25] M. Blaum, J. Brady, J. Bruck, and J. Menon, “Evenodd: An efficient scheme for tolerating double disk failures in raid architectures,” IEEE Trans. Comput., vol.44, no.2, pp.192-202, 1995. 10.1109/12.364531 – reference: [31] J. Li, S. Yang, X. Wang, X. Xue, and B. Li, “Tree-structured data regeneration with network coding in distributed storage systems,” Proc. 17th International Workshop on Quality of Service (IWQoS), 2009, pp.1-9, 2009. 10.1109/iwqos.2009.5201391 – reference: [3] D. Borthakur, “The hadoop distributed file system: Architecture and design,” Hadoop Project Website, vol.11, p.21, 2007. – reference: [22] M.G. Luby, M. Mitzenmacher, M.A. Shokrollahi, and D.A. Spielman, “Improved low-density parity-check codes using irregular graphs,” IEEE Trans. Inf. Theory, vol.47, no.2, pp.585-598, 2001. 10.1109/18.910576 – reference: [19] R.L. Collins and J.S. Plank, “Assessing the performance of erasure codes in the wide-area,” International Conference on Dependable Systems and Networks, Yokohama, Japan, IEEE, pp.182-187, 2005. 10.1109/dsn.2005.21 – ident: 18 doi: 10.1145/258533.258573 – ident: 25 doi: 10.1109/12.364531 – ident: 43 – ident: 12 doi: 10.1145/258533.258660 – ident: 32 doi: 10.1109/CC.2013.6506936 – ident: 24 doi: 10.1109/TIT.2006.874390 – ident: 30 doi: 10.1109/TIT.2010.2054295 – ident: 35 doi: 10.1007/978-3-642-28145-7_35 – ident: 10 – ident: 26 doi: 10.1109/18.746809 – ident: 19 doi: 10.1109/DSN.2005.21 – ident: 22 doi: 10.1109/18.910576 – ident: 31 doi: 10.1109/IWQoS.2009.5201391 – ident: 28 – ident: 11 doi: 10.1147/rd.111.0025 – ident: 27 doi: 10.1109/TC.2007.70830 – ident: 15 doi: 10.1016/j.future.2011.08.005 – ident: 40 doi: 10.1007/3-540-49543-6_14 – ident: 36 doi: 10.1109/NCA.2008.45 – ident: 37 doi: 10.1109/SNAPI.2010.16 – ident: 7 – ident: 41 – ident: 17 – ident: 42 – ident: 3 – ident: 14 doi: 10.1007/11558989_21 – ident: 39 doi: 10.1016/j.camwa.2006.05.002 – ident: 5 – ident: 33 doi: 10.1109/IPDPS.2008.4536322 – ident: 34 doi: 10.1109/PDSW.2008.4811887 – ident: 38 – ident: 2 doi: 10.1145/1165389.945450 – ident: 16 doi: 10.1007/s10115-013-0638-6 – ident: 4 doi: 10.1145/301250.301441 – ident: 6 – ident: 8 – ident: 20 doi: 10.1109/DSN.2006.40 – ident: 1 doi: 10.3724/SP.J.1001.2012.04175 – ident: 9 doi: 10.1145/1713072.1713075 – ident: 21 – ident: 23 doi: 10.1109/ISIT.2004.1365074 – ident: 13 doi: 10.1007/3-540-45748-8_31 – ident: 29 doi: 10.1109/TIT.2011.2159049 |
| SSID | ssj0018215 |
| Score | 2.1400557 |
| Snippet | Erasure codes have been considered as one of the most promising techniques for data reliability enhancement and storage efficiency in modern distributed... |
| SourceID | unpaywall proquest crossref jstage |
| SourceType | Open Access Repository Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 627 |
| SubjectTerms | Algorithms Codes Coding data allocation erasure codes Microprocessors multiple threads Parallel processing parallelism performance evaluation storage Storage systems Synchronism thread affinity |
| Title | An Efficient Parallel Coding Scheme in Erasure-Coded Storage Systems |
| URI | https://www.jstage.jst.go.jp/article/transinf/E101.D/3/E101.D_2017EDP7218/_article/-char/en https://www.proquest.com/docview/2014565424 https://www.jstage.jst.go.jp/article/transinf/E101.D/3/E101.D_2017EDP7218/_pdf |
| UnpaywallVersion | publishedVersion |
| Volume | E101.D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| ispartofPNX | IEICE Transactions on Information and Systems, 2018/03/01, Vol.E101.D(3), pp.627-643 |
| journalDatabaseRights | – providerCode: PRVAFT databaseName: Open Access Digital Library customDbUrl: eissn: 1745-1361 dateEnd: 99991231 omitProxy: true ssIdentifier: ssj0018215 issn: 0916-8532 databaseCode: KQ8 dateStart: 20080101 isFulltext: true titleUrlDefault: http://grweb.coalliance.org/oadl/oadl.html providerName: Colorado Alliance of Research Libraries |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1Rb9MwED5tHRLjgcEAURiTH3hNssSO7TxWa6eJiaoIKo2nyHHswihp1KVC8Os5N07oeAAh8RRHuYts39n3neW7A3htS10wllFU3kwEzB00KX2mAp3YLLXiDDG6C05-O-WXc_bmOr3eg2kXC-OuVd4gLloY9wgXq_CmjvwkRo3bvnHiowkqUjiOqG_kaMXEZDxDb0ZGeV3afTjgKWLzARzMp7PRx23CvZgHaJuSNkIyDWLKYx9Ll0rR_9vd9hKmrN2_7tiqe2237iDR-5uqVt-_qeVyxyhdHMGqG057F-VLuGmKUP_4LdPj_xvvI3jo8SsZtbyPYc9Ux3DU1YYgfqs4hgc7iQ6fwHhUkck2WQXaODJTa1fBZUnOV85yItMn89WQz0iz3p5YBvjBlOR9g_q5MMRnVX8K84vJh_PLwNdvCDTjvMEFWBS2ZDwxEmGR0WXG0B9JMq651kwU0giEC0pRdElFVrpS7gi3Mpu6SqS0FPQZDKpVZZ4DkRYVR8YizqxChZLKUKYp57YolJZaDIF2ksq1T27uamwsc-fkoHzzbi53524IQc9Vt8k9_kJ_1cqqp_aS-kXtBJSPc9o1drh7YhdPh5vSEE46Tcr9xnHr6B3GZgkbQthr159757X1xb8yvIRDfJPtjboTGDTrjXmFEKspTmH_6p089avnJ5yAI74 |
| linkProvider | Unpaywall |
| linkToUnpaywall | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1Rb9MwED6NDonxwGCA6DaQH3hNssSO7TxWa6cJRFUJKo2nyHHswihp1KVC26_n3Dih4wGExFMc5S6yfWffd5bvDuCtLXXBWEZReTMRMHfQpPSZCnRis9SKM8ToLjj5w5Rfztm7q_RqD6ZdLIy7VnmNuGhh3CNcrMLrOvKTGDVu-8aJjyaoSOE4or6RoxUTk_EMvRkZ5XVpH8A-TxGbD2B_Pp2NPm8T7sU8QNuUtBGSaRBTHvtYulSK_t_utpcwZe3-dc9WPWy7dQ-JPtpUtbr9oZbLHaN0cQirbjjtXZRv4aYpQn33W6bH_zfep_DE41cyanmfwZ6pjuCwqw1B_FZxBI93Eh0-h_GoIpNtsgq0cWSm1q6Cy5Kcr5zlRKYv5rshX5FmvT2xDPCDKcnHBvVzYYjPqv4C5heTT-eXga_fEGjGeYMLsChsyXhiJMIio8uMoT-SZFxzrZkopBEIF5Si6JKKrHSl3BFuZTZ1lUhpKehLGFSryrwCIi0qjoxFnFmFCiWVoUxTzm1RKC21GALtJJVrn9zc1dhY5s7JQfnm3Vzuzt0Qgp6rbpN7_IX-fSurntpL6he1E1A-zmnX2OHuiV08HW5KQzjtNCn3G8eNo3cYmyVsCGGvXX_undfW439lOIEDfJPtjbpTGDTrjXmNEKsp3vh18xPLuCLJ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+Efficient+Parallel+Coding+Scheme+in+Erasure-Coded+Storage+Systems&rft.jtitle=IEICE+Transactions+on+Information+and+Systems&rft.au=LIU%2C+Guangming&rft.au=DONG%2C+Wenrui&rft.date=2018-03-01&rft.pub=The+Institute+of+Electronics%2C+Information+and+Communication+Engineers&rft.issn=0916-8532&rft.eissn=1745-1361&rft.volume=E101.D&rft.issue=3&rft.spage=627&rft.epage=643&rft_id=info:doi/10.1587%2Ftransinf.2017EDP7218&rft.externalDocID=article_transinf_E101_D_3_E101_D_2017EDP7218_article_char_en |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0916-8532&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0916-8532&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0916-8532&client=summon |