Summary
Business intelligence (BI) refers to the processes and technologies used to analyze business data and extract actionable insights to improve decision-making. It involves collecting, analyzing, and presenting data in a way that helps organizations understand their performance, identify trends, and make informed strategic and operational decisions. BI tools help transform raw data into meaningful information, often through dashboards, reports, and visualizations.
Source: Gemini AI Overview
OnAir Post: Business Intelligence
About
Key Aspects
- Data Collection and StorageBI systems gather data from various sources, both internal (e.g., sales, finance) and external (e.g., market data, social media). This data is often stored in a data warehouse or other repositories for analysis.
- Data AnalysisBI tools use various techniques, including statistical analysis, data mining, and predictive analytics, to analyze the data and uncover trends, patterns, and anomalies.
- Data VisualizationThe analyzed data is then presented in user-friendly formats like dashboards, reports, charts, and graphs to make it easy to understand and interpret.
- Actionable InsightsThe goal of BI is to provide insights that can be used to make better business decisions, improve performance, and gain a competitive advantage.
Source: Google Gemini Overview
Benefits
- Improved Decision-Making:
BI provides data-driven insights that support better strategic and operational decisions.
- Increased Efficiency:By automating data analysis and reporting, BI can save time and resources.
- Enhanced Performance:BI helps organizations track key performance indicators (KPIs), identify areas for improvement, and optimize their operations.
- Competitive Advantage:By understanding market trends and customer behavior, BI can help organizations gain a competitive edge.
- Better Forecasting:BI tools can be used to forecast future trends and make predictions based on historical data.
Source: Google Gemini Overview
Use in Data Engineering
- Data engineering provides the foundation by making data available for analysis and visualization.
- Visualization is a key component of BI, allowing users to easily understand and interpret data.
- BI utilizes both data engineering and visualization to derive insights and support decision-making.
In essence, data engineers build the systems, data visualization provides the tools to understand the data, and business intelligence leverages both to drive better business outcomes.
Challenges
Business intelligence (BI) in data engineering faces several key challenges, including data quality issues, integrating diverse data sources, ensuring data security and compliance, and managing scalability and performance. Additionally, a lack of skilled personnel, complex tools, and the need to keep up with rapidly changing technology also pose significant hurdles.
Initial Source for content: Gemini AI Overview
[Enter your questions, feedback & content (e.g. blog posts, Google Slide or Word docs, YouTube videos) on the key issues and challenges related to this post in the “Comment” section below. Post curators will review your comments & content and decide where and how to include it in this section.]
1. Data Quality and Consistency
- Problem
Poor data quality, including inaccuracies, inconsistencies, and incompleteness, is a major obstacle in BI. - Impact
Inaccurate data can lead to flawed insights and poor decision-making. - Solutions
Implementing data cleansing processes, establishing data governance policies, and using tools for data quality monitoring can help address this.
2. Integrating Diverse Data Sources
- ProblemOrganizations often have data scattered across various systems, formats, and locations.
- ImpactThis makes it difficult to gain a unified view of the business and can hinder effective analysis.
- SolutionsEmploying ETL (Extract, Transform, Load) tools, building robust data models, and establishing data governance frameworks can help integrate data from different sources.
3. Data Security and Compliance
- ProblemProtecting sensitive data from unauthorized access and breaches is crucial, especially with regulations like GDPR and HIPAA.
- ImpactNon-compliance can result in significant penalties and reputational damage.
- SolutionsImplementing strong security measures, establishing access controls, and ensuring adherence to relevant regulations are essential.
4. Scalability and Performance
- ProblemAs data volumes grow, BI systems can experience performance issues and scalability limitations.
- ImpactSlow query times and limited capacity can hinder the ability to analyze large datasets effectively.
- SolutionsOptimizing database performance, using scalable infrastructure, and exploring cloud-based solutions can help address scalability challenges.
5. Skills Gap and Talent Shortages
- ProblemThere’s a growing demand for skilled data professionals, including data engineers and BI developers, but the supply of qualified candidates is often limited.
- ImpactThis can lead to delays in BI projects and hinder the organization’s ability to leverage data effectively.
- SolutionsInvesting in training programs, partnering with specialized recruitment agencies, and fostering a data-driven culture can help address the skills gap.
6. Organizational Alignment and User Adoption
- ProblemSuccessful BI implementation requires organizational alignment, change management, and user adoption.
- ImpactWithout proper buy-in from stakeholders and effective user training, BI initiatives may not deliver the intended value.
- SolutionsEstablishing clear communication channels, providing adequate training on BI tools, and fostering a data-driven culture can improve user adoption.
7. Keeping Up with New Technologies
- ProblemThe field of data engineering and BI is constantly evolving, with new technologies and tools emerging regularly.
- ImpactOrganizations need to adapt to these changes to remain competitive and leverage the latest advancements.
- SolutionsEncouraging continuous learning, participating in industry events, and exploring new technologies can help organizations stay ahead of the curve.
Research
Research in Data Engineering for Business Intelligence (BI) focuses on building the infrastructure and systems that enable the collection, storage, and processing of data to support informed decision-making. This involves designing, building, and maintaining data pipelines, data warehouses, and data lakes to ensure data is readily available and usable for BI analysis. Essentially, data engineering provides the foundation for BI by making raw data accessible and reliable for analysis and reporting.
In essence, data engineering is the critical infrastructure that underpins Business Intelligence, allowing businesses to leverage their data for improved performance and competitive advantage.
Initial Source for content: Gemini AI Overview
Key aspects of Data Engineering for BI
- Data Collection and IntegrationData engineers gather data from various sources, including databases, APIs, and applications, and integrate it into a centralized repository.
- Data is often stored in data warehouses or data lakes, which are designed to handle large volumes of structured and unstructured data.
- Data pipelines are automated workflows that move and transform data from source systems to the data warehouse or data lake.
- Data Quality and SecurityData engineers are responsible for ensuring the quality, security, and reliability of the data.
- Performance OptimizationData engineering also involves optimizing data storage and processing for performance and scalability.
- ETL ProcessesExtract, Transform, Load (ETL) processes are crucial for preparing data for BI analysis by extracting it from source systems, transforming it into a usable format, and loading it into a data warehouse or data lake.
How it supports Business Intelligence
- Actionable InsightsData engineering provides the foundation for BI by making data accessible and reliable, enabling analysts to derive actionable insights.
- Informed Decision-MakingBy providing access to clean and organized data, data engineering empowers businesses to make data-driven decisions.
- Performance Monitoring and OptimizationData engineering enables the tracking of key performance indicators (KPIs) and the identification of areas for improvement.
- Strategic PlanningBy providing a comprehensive view of business performance, data engineering supports strategic planning and long-term business goals.
Projects
Recent and future data engineering projects for business intelligence are focused on leveraging cloud-native solutions, real-time data processing, and AI/ML integration to enhance data accessibility, drive automation, and improve decision-making. Key areas include building robust ETL pipelines, data quality monitoring, and developing data mesh and data fabric architectures.
Initial Source for content: Gemini AI Overview
[Enter your questions, feedback & content (e.g. blog posts, Google Slide or Word docs, YouTube videos) on current and future projects implementing solutions to this post challenges in the “Comment” section below. Post curators will review your comments & content and decide where and how to include it in this section.]
Recent and Ongoing Projects
- Organizations are increasingly shifting to cloud platforms like Snowflake, Databricks, and Amazon Redshift for scalable and cost-effective data storage and processing.
- ETL Pipeline DevelopmentData engineers are building and optimizing ETL (Extract, Transform, Load) pipelines to ingest, clean, and transform data from various sources into data warehouses or data lakes for analysis.
- Tools like Apache Kafka and Apache Flink are used to build pipelines for real-time data analysis, enabling applications like fraud detection and personalized customer experiences.
- Projects are focused on implementing systems to monitor data quality, ensuring accuracy and reliability for downstream analytics and decision-making.
- Organizations are adopting these architectures to decentralize data ownership and management, enabling more agile and efficient data access across different teams.
- Data engineers are building pipelines and infrastructure to support AI/ML models, including data preparation, feature engineering, and model deployment.
- Data Governance and SecurityWith increasing regulations and data breaches, data engineers are implementing robust data governance policies and security measures within data pipelines.
Future Trends and Projects
- Zero-ETL ArchitecturesThe industry is moving towards architectures that minimize or eliminate the need for traditional ETL processes, potentially leveraging techniques like Change Data Capture (CDC) or streaming technologies.
- Formats like Apache Iceberg, Apache Hudi, and Delta Lake are gaining traction, offering benefits like transactional support and schema evolution for data lakes.
- Standardized data contracts are being implemented to improve data quality and collaboration between data producers and consumers.
- The use of synthetic data is growing as a way to address data privacy concerns and augment training datasets for machine learning models.
- Edge computing platforms are becoming increasingly important for real-time data processing in industries like manufacturing and remote monitoring.
- Integrating AI and machine learning into BI tools to automate data preparation, insight generation, and sharing is a growing trend.
- Data lakehouses, which combine the best features of data warehouses and data lakes, are becoming the default architecture for many organizations.
Wikipedia
Contents
Business intelligence (BI) consists of strategies, methodologies, and technologies used by enterprises for data analysis and management of business information.[1] Common functions of BI technologies include reporting, online analytical processing, analytics, dashboard development, data mining, process mining, complex event processing, business performance management, benchmarking, text mining, predictive analytics, and prescriptive analytics.
BI tools can handle large amounts of structured and sometimes unstructured data to help organizations identify, develop, and otherwise create new strategic business opportunities. They aim to allow for the easy interpretation of these big data. Identifying new opportunities and implementing an effective strategy based on insights is assumed to potentially provide businesses with a competitive market advantage and long-term stability, and help them take strategic decisions.[2]
Business intelligence can be used by enterprises to support a wide range of business decisions ranging from operational to strategic. Basic operating decisions include product positioning or pricing. Strategic business decisions involve priorities, goals, and directions at the broadest level. In all cases, Business Intelligence (BI) is considered most effective when it combines data from the market in which a company operates (external data) with data from internal company sources, such as financial and operational information. When integrated, external and internal data provide a comprehensive view that creates ‘intelligence’ not possible from any single data source alone.[3]
Among their many uses, business intelligence tools empower organizations to gain insight into new markets, to assess demand and suitability of products and services for different market segments, and to gauge the impact of marketing efforts.[4]
BI applications use data gathered from a data warehouse (DW) or from a data mart, and the concepts of BI and DW combine as "BI/DW"[5] or as "BIDW". A data warehouse contains a copy of analytical data that facilitates decision support.
History
The earliest known use of the term business intelligence is in Richard Millar Devens' Cyclopædia of Commercial and Business Anecdotes (1865). Devens used the term to describe how the banker Sir Henry Furnese gained profit by receiving and acting upon information about his environment, prior to his competitors:
Throughout Holland, Flanders, France, and Germany, he maintained a complete and perfect train of business intelligence. The news of the many battles fought was thus received first by him, and the fall of Namur added to his profits, owing to his early receipt of the news.
— Devens, p. 210
The ability to collect and react accordingly based on the information retrieved, Devens says, is central to business intelligence.[6]
When Hans Peter Luhn, a researcher at IBM, used the term business intelligence in an article published in 1958, he employed the Webster's Dictionary definition of intelligence: "the ability to apprehend the interrelationships of presented facts in such a way as to guide action towards a desired goal."[7]
In 1989, Howard Dresner (later a Gartner analyst) proposed business intelligence as an umbrella term to describe "concepts and methods to improve business decision making by using fact-based support systems."[8] It was not until the late 1990s that this usage was widespread.[9]
Definition
According to Solomon Negash and Paul Gray, business intelligence (BI) can be defined as systems that combine:
with analysis to evaluate complex corporate and competitive information for presentation to planners and decision makers, with the objective of improving the timeliness and the quality of the input to the decision process."[10]
According to Forrester Research, business intelligence is "a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision-making."[11] Under this definition, business intelligence encompasses information management (data integration, data quality, data warehousing, master-data management, text- and content-analytics, et al.). Therefore, Forrester refers to data preparation and data usage as two separate but closely linked segments of the business-intelligence architectural stack.
Some elements of business intelligence are:[citation needed]
- Multidimensional aggregation and allocation
- Denormalization, tagging, and standardization
- Realtime reporting with analytical alert
- A method of interfacing with unstructured data sources
- Group consolidation, budgeting, and rolling forecasts
- Statistical inference and probabilistic simulation
- Key performance indicators optimization
- Version control and process management
- Open item management
Forrester distinguishes this from the business-intelligence market, which is "just the top layers of the BI architectural stack, such as reporting, analytics, and dashboards."[12]
Compared with competitive intelligence
Though the term business intelligence is sometimes a synonym for competitive intelligence (because they both support decision making), BI uses technologies, processes, and applications to analyze mostly internal, structured data and business processes while competitive intelligence gathers, analyzes, and disseminates information with a topical focus on company competitors. If understood broadly, competitive intelligence can be considered as a subset of business intelligence.[13]
Compared with business analytics
Business intelligence and business analytics are sometimes used interchangeably, but there are alternate definitions.[14] Thomas Davenport, professor of information technology and management at Babson College argues that business intelligence should be divided into querying, reporting, Online analytical processing (OLAP), an "alerts" tool, and business analytics. In this definition, business analytics is the subset of BI focusing on statistics, prediction, and optimization, rather than the reporting functionality.[15]
Unstructured data
Business operations can generate a very large amount of data in the form of e-mails, memos, notes from call-centers, news, user groups, chats, reports, web-pages, presentations, image-files, video-files, and marketing material. According to Merrill Lynch, more than 85% of all business information exists in these forms; a company might only use such a document a single time.[16] Because of the way it is produced and stored, this information is either unstructured or semi-structured.
The management of semi-structured data is an unsolved problem in the information technology industry.[17] According to projections from Gartner (2003), white-collar workers spend 30–40% of their time searching, finding, and assessing unstructured data. BI uses both structured and unstructured data. The former is easy to search, and the latter contains a large quantity of the information needed for analysis and decision-making.[17][18] Because of the difficulty of properly searching, finding, and assessing unstructured or semi-structured data, organizations may not draw upon these vast reservoirs of information, which could influence a particular decision, task, or project. This can ultimately lead to poorly informed decision-making.[16]
Therefore, when designing a business intelligence/DW-solution, the specific problems associated with semi-structured and unstructured data must be accommodated for as well as those for the structured data.
Limitations of semi-structured and unstructured data
There are several challenges to developing BI with semi-structured data. According to Inmon & Nesavich,[19] some of those are:
- Physically accessing unstructured textual data – unstructured data is stored in a huge variety of formats.
- Terminology – Among researchers and analysts, there is a need to develop standardized terminology.
- Volume of data – As stated earlier, up to 85% of all data exists as semi-structured data. Couple that with the need for word-to-word and semantic analysis.
- Searchability of unstructured textual data – A simple search on some data, e.g. apple, results in links where there is a reference to that precise search term. (Inmon & Nesavich, 2008)[19] gives an example: "a search is made on the term felony. In a simple search, the term felony is used, and everywhere there is a reference to felony, a hit to an unstructured document is made. But a simple search is crude. It does not find references to crime, arson, murder, embezzlement, vehicular homicide, and such, even though these crimes are types of felonies".
Metadata
To solve problems with searchability and assessment of data, it is necessary to know something about the content. This can be done by adding context through the use of metadata.[16][needs independent confirmation] Many systems already capture some metadata (e.g. filename, author, size, etc.), but more useful would be metadata about the actual content – e.g. summaries, topics, people, or companies mentioned. Two technologies designed for generating metadata about content are automatic categorization and information extraction.
Generative AI
Generative business intelligence is the application of generative AI techniques, such as large language models, in business intelligence. This combination facilitates data analysis and enables users to interact with data more intuitively, generating actionable insights through natural language queries. Microsoft Copilot was for example integrated into the business analytics tool Power BI.[20]
Applications
Business intelligence can be applied to the following business purposes:
- Performance metrics and benchmarking inform business leaders of progress towards business goals.[21] (Business process management).[citation needed]
- Analytics quantify processes for a business to arrive at optimal decisions, and to perform business knowledge discovery. Analytics may variously involve data mining, process mining, statistical analysis, predictive analytics, predictive modeling, business process modeling, data lineage, complex event processing, and prescriptive analytics. For example within banking industry, academic research has explored potential for BI based analytics in credit evaluation, customer churn management for managerial adoption[22]
- Reporting, dashboards and data visualization,[21] executive information system, and/or OLAP
- BI can facilitate collaboration both inside and outside the business by enabling data sharing and electronic data interchange[21]
- Knowledge management is concerned with the creation, distribution, use, and management of business intelligence, and of business knowledge in general.[21] Knowledge management leads to learning management and regulatory compliance.[citation needed]
Roles
Some common technical roles for business intelligence developers are:[23]
Risk
In a 2013 report, Gartner categorized business intelligence vendors as either an independent "pure-play" vendor or a consolidated "mega-vendor".[24][non-primary source needed] In 2019, the BI market was shaken within Europe for the new legislation of GDPR (General Data Protection Regulation) which puts the responsibility of data collection and storage onto the data user with strict laws in place to make sure the data is compliant. Growth within Europe has steadily increased since May 2019 when GDPR was brought. The legislation refocused companies to look at their own data from a compliance perspective but also revealed future opportunities using personalization and external BI providers to increase market share.[25]
See also
- Agile Business Intelligence
- Analytic applications
- Arcplan
- Artificial intelligence marketing
- Business activity monitoring
- Business Intelligence 2.0
- Business Intelligence Competency Center
- Business intelligence software
- Business process discovery
- Business process management
- Customer dynamics
- Decision engineering
- Embedded analytics
- Enterprise planning systems
- Integrated business planning
- Management information system
- Mobile business intelligence
- Operational intelligence
- Process mining
- Real-time business intelligence
- Sales intelligence
- Test and learn
References
- ^ Dedić N. & Stanier noC. (2016). "Measuring the Success of Changes to Existing Business Intelligence Solutions to Improve Business Intelligence Reporting" (PDF). Measuring the Success of Changes to Existing Business Intelligence Solutions to Improve Business Intelligence Reporting. Lecture Notes in Business Information Processing. Vol. 268. Springer International Publishing. pp. 225–236. doi:10.1007/978-3-319-49944-4_17. ISBN 978-3-319-49943-7. S2CID 30910248.
- ^ (Rud, Olivia (2009). Business Intelligence Success Factors: Tools for Aligning Your Business in the Global Economy. Hoboken, N.J.: Wiley & Sons. ISBN 978-0-470-39240-9.)
- ^ Coker, Frank (2014). Pulse: Understanding the Vital Signs of Your Business. Ambient Light Publishing. pp. 41–42. ISBN 978-0-9893086-0-1.
- ^ Chugh, R. & Grandhi, S. (2013,). "Why Business Intelligence? Significance of Business Intelligence tools and integrating BI governance with corporate governance". International Journal of E-Entrepreneurship and Innovation', vol. 4, no.2, pp. 1–14.
- ^ Golden, Bernard (2013). Amazon Web Services For Dummies. John Wiley & Sons. p. 234. ISBN 9781118652268. Retrieved 6 July 2014.
[...] traditional business intelligence or data warehousing tools (the terms are used so interchangeably that they're often referred to as BI/DW) are extremely expensive [...]
- ^ Miller Devens, Richard (1865). Cyclopaedia of Commercial and Business Anecdotes; Comprising Interesting Reminiscences and Facts, Remarkable Traits and Humors of Merchants, Traders, Bankers Etc. in All Ages and Countries. D. Appleton and company. p. 210. Retrieved 15 February 2014.
business intelligence.
- ^ Luhn, H. P. (1958). "A Business Intelligence System" (PDF). IBM Journal of Research and Development. 2 (4): 314–319. doi:10.1147/rd.24.0314. Archived from the original (PDF) on 13 September 2008.
- ^ D. J. Power (10 March 2007). "A Brief History of Decision Support Systems, version 4.0". DSSResources.COM. Retrieved 10 July 2008.
- ^ Power, D. J. "A Brief History of Decision Support Systems". Retrieved 1 November 2010.
- ^ Springer-Verlag Berlin Heidelberg, Springer-Verlag Berlin Heidelberg (21 November 2008). Topic Overview: Business Intelligence. doi:10.1007/978-3-540-48716-6. ISBN 978-3-540-48715-9.
- ^ Evelson, Boris (21 November 2008). "Topic Overview: Business Intelligence".
- ^ Evelson, Boris (29 April 2010). "Want to know what Forrester's lead data analysts are thinking about BI and the data domain?". Archived from the original on 6 August 2016. Retrieved 4 November 2010.
- ^ Kobielus, James (30 April 2010). "What's Not BI? Oh, Don't Get Me Started... Oops Too Late... Here Goes..." Archived from the original on 7 May 2010. Retrieved 4 November 2010.
"Business" intelligence is a non-domain-specific catchall for all the types of analytic data that can be delivered to users in reports, dashboards, and the like. When you specify the subject domain for this intelligence, then you can refer to "competitive intelligence", "market intelligence", "social intelligence", "financial intelligence", "HR intelligence", "supply chain intelligence", and the like.
- ^ "Business Analytics vs Business Intelligence?". timoelliott.com. 9 March 2011. Retrieved 15 June 2014.
- ^ Henschen, Doug (4 January 2010). "Analytics at Work: Q&A with Tom Davenport" (Interview). Archived from the original on 3 April 2012. Retrieved 26 September 2011.
- ^ a b c Rao, R. (2003). "From unstructured data to actionable intelligence" (PDF). IT Professional. 5 (6): 29–35. doi:10.1109/MITP.2003.1254966.
- ^ a b Blumberg, R. & S. Atre (2003). "The Problem with Unstructured Data" (PDF). DM Review: 42–46. Archived from the original (PDF) on 25 January 2011.
- ^ Negash, S (2004). "Business Intelligence". Communications of the Association for Information Systems. 13: 177–195. doi:10.17705/1CAIS.01315.
- ^ a b Inmon, B. & A. Nesavich, "Unstructured Textual Data in the Organization" from "Managing Unstructured data in the organization", Prentice Hall 2008, pp. 1–13
- ^ Novet, Jordan (23 May 2023). "Microsoft is bringing an A.I. chatbot to data analysis". CNBC. Retrieved 19 August 2024.
- ^ a b c d Feldman, D.; Himmelstein, J. (2013). Developing Business Intelligence Apps for SharePoint. O'Reilly Media, Inc. pp. 140–1. ISBN 9781449324681. Retrieved 8 May 2018.
- ^ Moro, Sérgio; Cortez, Paulo; Rita, Paulo (February 2015). "Business intelligence in banking: A literature analysis from 2002 to 2013 using text mining and latent Dirichlet allocation". Expert Systems with Applications. 42 (3): 1314–1324. doi:10.1016/j.eswa.2014.09.024. hdl:10071/8522. S2CID 15595226.
- ^ Roles in data - Learn | Microsoft Docs
- ^ Andrew Brust (14 February 2013). "Gartner releases 2013 BI Magic Quadrant". ZDNet. Retrieved 21 August 2013.
- ^ SaaS BI growth will soar in 2010. InfoWorld (1 February 2010). Retrieved 17 January 2012.
Bibliography
- Kimball, Ralph; et al. (1998). The Data warehouse Lifecycle Toolkit" (2nd ed.). John Wiley & Sons Inc. ISBN 0-470-47957-4.
- Rausch, Peter; Sheta, Alaa; Ayesh, Aladdin (2013). Business Intelligence and Performance Management: Theory, Systems, and Industrial Applications. Springer Verlag U.K. ISBN 978-1-4471-4865-4..
- Munoz, J.M. (2017). Global Business Intelligence. Routledge : UK. ISBN 978-1-1382-03686.
- Chaudhuri, Surajit; Dayal, Umeshwar; Narasayya, Vivek (August 2011). "An Overview of Business Intelligence Technology". Communications of the ACM. 54 (8): 88–98. doi:10.1145/1978542.1978562. S2CID 13843514.