Apache Flume: Distributed Log Collection for Hadoop by Steve Hoffman

By Steve Hoffman

circulation facts to Hadoop utilizing Apache Flume


  • Integrate Flume along with your information sources
  • Transcode your information en-route in Flume
  • Route and separate your information utilizing typical expression matching
  • Configure failover paths and load-balancing to take away unmarried issues of failure
  • Utilize Gzip Compression for documents written to HDFS

In Detail

Apache Flume is a allotted, trustworthy, and on hand provider for successfully amassing, aggregating, and relocating quite a lot of log information. Its major aim is to carry information from purposes to Apache Hadoop's HDFS. It has an easy and versatile structure according to streaming information flows. it's powerful and fault tolerant with many failover and restoration mechanisms.

Apache Flume: disbursed Log assortment for Hadoop covers issues of HDFS and streaming data/logs, and the way Flume can unravel those difficulties. This publication explains the generalized structure of Flume, along with relocating info to/from databases, NO-SQL-ish information shops, in addition to optimizing functionality. This ebook contains real-world eventualities on Flume implementation.

Apache Flume: dispensed Log assortment for Hadoop starts off with an architectural evaluate of Flume after which discusses every one part intimately. It courses you thru the whole set up technique and compilation of Flume.

It provide you with a heads-up on the way to use channels and channel selectors. for every architectural part (Sources, Channels, Sinks, Channel Processors, Sink teams, and so forth) some of the implementations could be coated intimately in addition to configuration ideas. you should use it to customise Flume for your particular wishes. There are guidelines given on writing customized implementations in addition that will assist you study and enforce them.

  • By the top, you need to be in a position to build a chain of Flume brokers to move your streaming information and logs out of your platforms into Hadoop in close to actual time.
  • What you are going to study from this book

    • Understand the Flume architecture
    • Download and set up open resource Flume from Apache
    • Discover whilst to take advantage of a reminiscence or file-backed channel
    • Understand and configure the Hadoop dossier approach (HDFS) sink
    • Learn the best way to use sink teams to create redundant facts flows
    • Configure and use quite a few resources for consuming data
    • Inspect info documents and path to varied or a number of locations in accordance with payload content
    • Transform facts en-route to Hadoop
    • Monitor your information flows


    A starter consultant that covers Apache Flume in detail.

    Who this publication is written for

    Apache Flume: dispensed Log assortment for Hadoop is meant for those that are liable for relocating datasets into Hadoop in a well timed and trustworthy demeanour like software program engineers, database directors, and information warehouse administrators.

    Show description

    Read or Download Apache Flume: Distributed Log Collection for Hadoop PDF

    Similar software development books

    Software Engineering for Modern Web Applications: Methodologies and Technologies

    As sleek enterprises migrate from older info architectures to new Web-based structures, the self-discipline of software program engineering is altering either by way of applied sciences and methodologies. there's a have to study this new frontier from either a theoretical and pragmatic viewpoint, and provide not just a survey of latest applied sciences and methodologies yet discussions of the applicability and pros/cons of every.

    Advances in Computers, Vol. 11

    In view that its first quantity in 1960, Advances in pcs has awarded special insurance of thoughts in and software program and in computing device thought, layout, and functions. It has additionally supplied individuals with a medium during which they could learn their matters in better intensity and breadth than that allowed by means of average magazine articles.

    Lean architecture for agile software development

    A growing number of Agile initiatives are trying to find architectural roots as they fight with complexity and scale - and they are looking light-weight how one can do it nonetheless looking? during this book the authors help you in finding your personal course Taking cues from Lean development, they may help steer your undertaking towards practices with longstanding song documents Up-front structure?

    Advances in Computers and Information in Engineering Research

    This publication sequence goals to trap advances in desktops and data in engineering study, specifically via researchers and individuals of ASME's pcs & details in Engineering (CIE) department. The books may be released in either conventional and book codecs. The sequence is targeting advances in computational equipment, algorithms, instruments, and approaches at the innovative of study and improvement as they've got advanced and/or were suggested over the past 3 to 5 annual CIE meetings.

    Additional info for Apache Flume: Distributed Log Collection for Hadoop

    Sample text

    Finally, you'll want to make sure that whatever mechanism is writing new files into your spooling directory creates unique filenames, such as adding a timestamp (and possibly more). Reusing a filename will confuse the source and your data may not be processed. As always, remember that restarts and errors will create duplicate events on any files in the spooling directory that are retransmitted due to not being marked as finished. [ 52 ] Chapter 5 Syslog sources Syslog has been around for decades and is often used as an operating system level mechanism for capturing and moving logs around systems.

    Here is a table of configuration parameters you can adjust from the default values: Key type Required Type String Default Yes capacity No int 100 transactionCapacity No int 100 byteCapacityBufferPercentage No int (percent) 20% byteCapacity No 80% of JVM Heap keep-alive No long (bytes) int memory 3 (seconds) The default capacity of this channel is 100 Events. capacity=200 Remember if you increase this value you may also have to increase your Java heap space using the -Xmx and optionally the -Xms parameters.

    You cannot set this lower than 1000 milliseconds. Checkpoint files are also roll based on volume of data written to them using the maxFileSize property. You can lower this value for low traffic channels if you want to try and save some disk space. Let's say your maximum file size is 50,000 bytes but your channel only writes 500 bytes a day, it would take 100 days to fill a single log. Let's say that you were on day #100 and 2000 bytes came in all at once. Some data would be written to the old file and a new file would get started with the overflow.

    Download PDF sample

    Rated 4.81 of 5 – based on 10 votes