Andrew Chan
Actuarial Engineer | AXIS, Risk Integrity, IFRS 17๐๐ฅ๐๐ฏ๐๐ญ๐ ๐๐จ๐ฎ๐ซ ๐๐๐ญ๐ฎ๐๐ซ๐ข๐๐ฅ ๐๐๐ฉ๐จ๐ซ๐ญ๐ข๐ง๐ ๐ฐ๐ข๐ญ๐ก ๐ ๐๐ญ๐ซ๐๐๐ฆ๐ฅ๐ข๐ง๐๐, ๐๐ฎ๐ญ๐จ๐ฆ๐๐ญ๐๐ ๐๐ฉ๐ฉ๐ซ๐จ๐๐๐ก ๐ญ๐จ ๐๐ ๐๐ ๐๐
Published Jan 4, 2016
Analysts can use Excel to do a lot of their day-to-day jobs, so most financial processes involve a lot of Excel workbooks. However, there are some common data issues behind all these Excel processes, and I will examine how databases can resolve the problems.
Excel Challenges
A single Excel workbook can link to multiple Excel workbooks, and each linked workbook can link to moreExcel workbooks. Data is duplicated in many Excel workbooks. There are alsoinstances in which users copy and paste data from one Excel workbook to another, so data may not always be consistent. It is very time-consuming for analysts to understandthe data flow and the financial processes.
Excel has to load all the data into memory before it can process the data, so it could be more scalable. e.g. it cantakea long time to process if it tries to look at data from tables that each have a few hundred thousand records. It would be even worse if you used a 32-bit edition of Excel.
Excel workbooks often have minimum security because there are just toomany Excel workbooks. It is also difficult to implement security when the data and processes (or calculations) are so tightly tied to each other in the Excel workbook, e.g. data is hard coded in an Excel formula.
MostExcel workbooks only contain monthly data. Ifyou needto do a five-year analysis, consolidating the data may be lengthy, especially when the consolidated dataset has more than 1 million records. The complexity will be escalated if you need to integrate multiple workbooks.
What is a database?
The database is very similar to the Excel workbook. Databases store data in tables (worksheet), and tables have records (rows) and fields (columns). Buta worksheet in an Excel workbook can only store one million rows, whereas tables in a database can store billions, trillion...records. There are other components in databases, e.g.
Databases also provide central security, backup and restore, performance tuning, and other vital signsmonitoring.
What are the benefits?
If all the data is centrally stored in a well-designed database, then there would be no duplicated data, anddata is always consistent. Another less obvious benefit is that the Excel workbooks can be better controlled because data is now decoupled from the Excel workbook. Analysts do not need to modify the Excel workbook; the Excel workbook is always the same unless the analysts enhance the processes or fix a bug.
Analysts can start their analysis whenever they want; they no longer need to look for, transform, or validate data; the data is alwaysready to use.
The financial processes should performmuch faster. If a new process takes too long to run, a database administrator can continuously monitor the process, identify the bottleneck and add any necessaryindex to speed up the process.
Data is much safer in a database because modern databases often come with many housekeeping utilities, ensuring all data is in good shape.
Databases allow analysts to deliverbusiness insightsto their senior management better and faster.
Help improve contributions
Mark contributions as unhelpful if you find them irrelevant or not valuable to the article. This feedback is private to you and wonโt be shared publicly.
Contribution hidden for you
This feedback is never shared publicly, weโll use it to show better contributions to everyone.
Great points Andrew. Using databases also allows for more flexibility with regards to integrating information from different data-points. It also reduces the chance of duplicated information. Many processes are data-driven including the use of strategy, and the use of queries (or software) that allows for analytics to be undertaken which would be more challenging if all information is stored using Excel.
Properly architected a more exacting mode of memory management, response time, presentation etc not limited by proprietory limitations
PHP and SYLK ...
I would argue that any system or process can and will be dangerous when it's not set up optimally. Andrew, you've talked about daisy chains of links in Excel workbooks, but an arguably similar problem applies when you have SQL queries in a datamart that refer to other queries in a data warehouse that refer to other queries in another database... It might not be as obvious since people who write SQL queries are a bit smarter than to create such unnecessary links, but the issues in Excel are only highlighted because the of the overall scale of usage - people who are trained to use Excel properly don't create external links, much less chains of workbooks that are linked.The trend that is slowly picking up these days is to leave the data in the source systems and instead, use PowerPivot in lieu of Access. As Mary has pointed out, PowerPivot is perhaps Microsoft's big step to slowly phase out Access and incorporate more database-like functionality into Excel. As far as I can tell, aside from the form / data entry side of Access, there is not much that the average Access database does that now cannot be replicated in Power Query and Power Pivot (or equivalent Excel 2016 Power BI tool names: Get and Transform?).As an Excel trainer / consultant primarily, most of our clients may have the raw data stored in a database attached to a core business system. However, to invest in the infrastructure and expertise (people are expensive!) to develop and maintain that database, for a small-medium business, represents poor value compared to using the new Power BI tools in Excel to work with that data. Of course you will still want the database there to maintain data integrity. However, most of your analysis, unless that data is in the tens of millions+ of records, can and should be done in Excel instead since it's more transparent*.While Power Query doesn't have quite the same power as a proper SQL database environment, it does what most businesses require (though you might need to research and code it up, rather than follow the GUI entirely).* Excel - more transparent than databases and SQL code? Yes, absolutely! The key thing is training - it is much easier to maintain and enforce a consistent spreadsheet standard than it is to train up a team in SQL or other programming language, whereupon they will be just skilled enough to as dangerous as they are in Excel. I've been saying for years - the only reason Excel is seen as being dangerous and problematic is because of a lack of appropriate training.
The reason so many people use Excel as their default is not just because they're more familiar with Excel (though that is a big part) -- it's that most database software is set up to give aggregate results, and are not very good when you have to do number-crunching beyond summing subsets selected on particular attributes. People don't like all those interim steps and would like to go right to the number-crunching.Michael Failor's remark on building Excel VBA applications is getting closer -- if people can learn how to suck in the data they need from the databases (so they don't mess up the raw data) to do the number-crunching, that would improve processes immensely.FWIW, Microsoft has been trying to merge Excel & Access for years (I think Powerpivot is one of their steps for that), but perhaps they've given up.
To view or add a comment, sign in
More articles by this author
No more previous content
-
From Coder to Developer: Elevating Actuarial Software Practices
Mar 27, 2024
-
Mastering the Power of Modern Integrated Development Environments (IDEs)
Mar 26, 2024
-
How Libraries and Frameworks Revolutionized Software Development: A Shift Toward Collaborative Innovation
Mar 25, 2024
-
The Evolution of Programming Languages: From Binary to Low-Code Platforms
Mar 24, 2024
-
From Actuary to Data Engineering Expertise: Navigating the Transition
Mar 20, 2024
-
Transitioning from Actuary to Data Scientist: A Comprehensive Guide
Mar 19, 2024
-
Empowering Actuaries in a Digital Age: The Comprehensive Power of Power Platform
Mar 16, 2024
-
Unleashing Efficiency in Actuarial Processes: The Power of Power Query
Mar 9, 2024
-
Navigating Software Development Methodologies for Actuarial Model Development
Mar 2, 2024
-
From Excel to Low-Code: Ensuring We Don't Repeat Our Mistakes in the Next Generation of Actuarial Tools
Mar 1, 2024
No more next content
Sign in
Stay updated on your professional world
By clicking Continue, you agree to LinkedInโs User Agreement, Privacy Policy, and Cookie Policy.
New to LinkedIn? Join now
Insights from the community
-
Information Architecture
What are the best practices for designing a user-friendly database?
-
Database Development
What is the best index type for optimizing database queries?
-
Business Intelligence
How can you quickly troubleshoot database query errors?
-
Database Administration
What are the best practices for using query auditing to identify database normalization issues?
-
Database Administration
What are the best practices for querying tables with millions of rows?
-
Database Administration
How do you design a database for different user roles?
-
Database Administration
You're starting a new project and need to design a database. What are the most important things to know?
-
Software Design
How do you select data types for database tables?
-
Performance Tuning
How do you use indexes to optimize database queries?
-
Database Administration
What is a temporary table, and how can you use it to improve database queries?
Others also viewed
-
Excel is not a database
Brett Long 8mo
-
Key Differences Between CTEs, Temporary Tables, and Views
MOHIT WAGISH 1mo
-
Query optimization
Sandeep Sinha Mahapatra 10mo
-
How Do SQL Database Engines Work?
Vaste Uwase 7mo
-
What are databases, and why might I need a database?
LocalTeam Australia 1y
-
The Power of SQL Databases: Enhancing Real-Life Applications, Optimization Techniques, and Overcoming Limitations
Sarthak Dhawan 9mo
-
Advanced Analytics database Tools
V.S GOPTHREYA 11mo
-
Database Performance Tips
Pedro Mendes 5y
-
How to optimize database performance with advanced SQL Techniques
Isha Taneja 4mo
-
SQL Statements and the importance of Database design in building data integrity.
Innocent NHARAUNDA 3y