Skip to content

Commit

Permalink
updates
Browse files Browse the repository at this point in the history
  • Loading branch information
@ogbinar
ogbinar committed Feb 11, 2024
1 parent 1dc55f6 commit b9d1612
Show file tree
Hide file tree
Showing 4 changed files with 338 additions and 111 deletions.
36 changes: 30 additions & 6 deletions community.qmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,15 @@ An online event with key thought leaders and content creators in the Filipino te
{{< video https://youtu.be/ivzmUPRqxQ8?si=c_OdyC4FvqOJv6Bq title="Shifting to a Career in Data | RUG-PH 120" >}}
A panel discussion hosted by R User's Group Philippines on transitioning to a career in data.

[![Introduction to Big Data and Analytics](images/video-placeholder.png){width=100%}](https://www.facebook.com/watch/live/?ref=search&v=2388014311349277)
Kyle Escosia's talk on Big Data and Analytics, recorded live at an AWS Siklab Pilipinas event.

[![Building a Serverless Data Lake in AWS](images/video-placeholder.png){width=100%}](https://www.facebook.com/watch/live/?ref=watch_permalink&v=684903669389073)
A session by Kyle Escosia on creating a serverless data lake in AWS.

{{< video https://www.youtube.com/watch?v=Y9060t-BG0g title="Kwentuhan Nights (February 8, 2024)" >}}
The "Kwentuhan Meetup" is an online event for members of the Data Engineering Pilipinas group, hosted on Discord.

## Doc Ligot Interviews

### Kuya Dev
Expand All @@ -44,13 +53,21 @@ A panel discussion hosted by R User's Group Philippines on transitioning to a ca
{{< video https://www.youtube.com/watch?v=GZcYyILg3kc title="From Dropout to Tech Star: Aemy Obinguar's Inspirational Tale" >}}
### Sandy Lauguico
{{< video https://www.youtube.com/watch?v=8pJMFi3kIfQ title="Exclusive Interview: Sandy Lauguico's Data Engineering Transition" >}}
## Recorded Events and Talks

[![Introduction to Big Data and Analytics](images/video-placeholder.png){width=100%}](https://www.facebook.com/watch/live/?ref=search&v=2388014311349277)
Kyle Escosia's talk on Big Data and Analytics, recorded live at an AWS Siklab Pilipinas event.
### Neil Bacon's Transition: Discover how Leoneil Bacon moved from Biology to Data.
{{< video https://www.youtube.com/watch?v=Qq6hbQuf-1I title="Breaking Barriers: How Leoneil Bacon Switched from Biology to Data" >}}

[![Building a Serverless Data Lake in AWS](images/video-placeholder.png){width=100%}](https://www.facebook.com/watch/live/?ref=watch_permalink&v=684903669389073)
A session by Kyle Escosia on creating a serverless data lake in AWS.
### Trisha Nicdao's Journey: Insights into Trisha's shift from ESG to Data Analytics, touching on sustainability and GenZ perspectives.
{{< video https://www.youtube.com/watch?v=dViAWAVGGSA title="Empowering Change: Trisha Nicdao's Transition from ESG to Data Analytics" >}}

### Angel Felismino's Career Path: From Computer Science to leading in tech talent acquisition.
{{< video https://www.youtube.com/watch?v=XR4KNGA_Ipc title="From Code to Careers: Angel Felismino's Tech Talent Recruitment Journey" >}}

### Emmanuel Irog-Irog's Innovations: Discussing RAG and LLMs with a focus on recent projects.
{{< video https://www.youtube.com/watch?v=DydcJGyt7Xc title="The Core: Emmanuel Irog-Irog Explores Retrieval Augmented Generation" >}}

### Jerel John Velarde's Vision: Using data for enhancing democracy in AI startups.
{{< video https://www.youtube.com/watch?v=O1JXjUYwWc8 title="Jerel John Velarde's Mission in AI Startups for Democracy" >}}


## Blogs and Articles
Expand All @@ -68,4 +85,11 @@ A session by Kyle Escosia on creating a serverless data lake in AWS.

## People and Pages

- [Kyle Escosia](https://linktr.ee/klescosia) - Explore the work of Kyle Escosia, a Data Engineer with a passion for all things data.
- [Doc Ligot](https://docligot.com/)
- [Sherwin Pelayo](https://www.linkedin.com/in/sherwinpelayo/)
- [Kyle Escosia](https://linktr.ee/klescosia)
- [Sandy C. Lauguico](https://www.facebook.com/sandy.c.lauguico)
- [Josh Dev](https://www.facebook.com/profile.php?id=100087019650476)
- [Kuya Dev](https://www.facebook.com/KuyaDevDotCom)
- [Dev Stuff with JP](https://www.facebook.com/devstuffwithjp)
- [Alex Gamboa](https://www.facebook.com/alexgamboa34/)
152 changes: 114 additions & 38 deletions content/data-engineering-101.qmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,41 +6,117 @@ description: |

![Data Engineering Domain](../images/1692040311087.png)

## Data Engineering
- **Primary Focus**: Data engineering focuses on the practical aspects of data collection, data transformation, and data storage, preparing data for analytical or operational use.
- **Key Responsibilities**:
- Building and maintaining data architecture (databases, large-scale processing systems).
- Developing and managing data pipelines.
- Ensuring data availability and usability for data scientists and analysts.
- **Skills and Tools**:
- Programming languages (Python, Java, Scala).
- Database languages (SQL).
- Tools and frameworks (Apache Hadoop, Apache Spark, ETL tools, data warehousing solutions).

## Related Fields

### 1. Data Analysis
- Involves extracting insights from data.
- Tools: Excel, SQL, R, Python, BI tools (like Tableau, Power BI).

### 2. Data Science
- Encompasses data analysis, predictive modeling, and machine learning.
- Tools: Python, R, TensorFlow, machine learning libraries.

### 3. Machine Learning Engineering
- Focuses on building systems that learn from data.
- Tools: Python, machine learning frameworks, cloud computing platforms.

### 4. Business Intelligence (BI)
- Analyzing data to aid business decision-making.
- Tools: SQL, BI platforms (Tableau, Power BI, Looker).

### 5. Database Administration
- Managing and maintaining databases.
- Tools: SQL, database management systems (MySQL, PostgreSQL).

### 6. Big Data
- Handling large and complex data sets.
- Tools: Hadoop, Spark, NoSQL databases.

Each field plays a unique role in the data ecosystem, often collaborating to turn data into actionable insights. As the name suggests, our community focuses on all data career paths with emphasis on data engineering.
# Introduction to Data Engineering

Data engineering is a vital field in the landscape of data analytics and science. It lays the foundation for all data operations, from collecting and storing to processing and using data.

## What is Data Engineering?

- **Primary Focus**: Data engineering prepares data for analytical or operational use, emphasizing the practical application of data collection, transformation, and storage.

## Roles and Responsibilities

As a data engineer, you will be responsible for:

- Building and maintaining the infrastructure for data generation, collection, and distribution.
- Developing robust and scalable data pipelines that transform and transport data across systems.
- Ensuring data is readily available and in a usable format for analysts and data scientists to perform their tasks.

## Skills and Tools for Data Engineering

To thrive in data engineering, you will need to develop skills in:

- **Programming**: Become proficient in languages like Python, Java, or Scala.
- **Data Management & Governance**: Learn to manipulate databases using SQL.
- **Data Processing Frameworks**: Gain expertise in tools such as Apache Hadoop and Apache Spark.
- **Data Storage and Warehousing**: Understand how to implement and manage large-scale data storage solutions.

## Related Disciplines

Data engineering intersects with several related fields:

### Data Analysis
- **Description**: Extracting insights and making sense of data.
- **Tools**: Familiarize yourself with Excel, SQL, and BI tools like Tableau and Power BI.

### Data Science
- **Description**: Going beyond analysis to predict future trends and behaviors using data.
- **Tools**: Learn Python, R, and machine learning libraries to build predictive models.

### Machine Learning Engineering
- **Description**: Specializing in algorithms that can learn from and make decisions based on data.
- **Tools**: Master Python and frameworks like TensorFlow.

### Business Intelligence (BI)
- **Description**: Transforming data into actionable intelligence for business decisions.
- **Tools**: Use SQL and BI platforms like Tableau, Power BI, or Looker.

### Database Administration
- **Description**: Focusing on the technical management of database systems.
- **Tools**: Understand database management systems like MySQL and PostgreSQL.

### Big Data
- **Description**: Working with exceptionally large or complex data sets that require specialized approaches.
- **Tools**: Learn to work with Hadoop, Spark, and NoSQL databases.

## The Data Engineering Lifecycle

Understanding the Data Engineering Lifecycle is crucial for managing data effectively:

- **Generation**: Where and how data is produced.
- **Ingestion**: Moving data to a place where it can be used.
- **Transformation**: Converting data to a useful format.
- **Serving**: Making data accessible for use.
- **Storage**: Keeping data safe and retrievable.

## Outcomes of the Data Engineering Process

The end goal of data engineering can be one of the following:

- **Analytics**: Deriving insights that inform business strategies.
- **Machine Learning**: Training models to predict and act upon data.
- **Reverse ETL**: Integrating processed data back into operational systems.

## Supporting Practices in Data Engineering

These are the undercurrents that ensure the data flows smoothly throughout the lifecycle:

- **Security**: Protecting data integrity and privacy.
- **Data Management**: Ensuring that data is organized and maintained properly.
- **DataOps**: Streamlining the collaboration between teams working with data.
- **Data Architecture**: Creating the blueprint for data collection and usage.
- **Orchestration**: Automating processes and workflows.
- **Software Engineering**: Developing the applications that handle data.

Becoming a data engineer means you'll be at the intersection of data, technology, and business, ensuring that data is a valuable asset that can be leveraged to its full potential.

# Career Opportunities with Data Engineering Skills

Data engineering skills can lead to diverse career paths. Here are some potential titles and roles:

## Alternative Careers

- **Data Analyst**: Analyzes data to help inform business decisions.
- **Machine Learning Engineer**: Creates algorithms to predict patterns and behaviors.
- **Database Administrator**: Manages and maintains database systems.
- **Business Intelligence Analyst**: Converts data into actionable business insights.
- **Data Architect**: Designs and manages data solutions.
- **Data Science Generalist**: Handles various data-related tasks in smaller companies.
- **Systems Analyst**: Improves IT systems through data analysis.
- **Product Manager**: Integrates data insights into product strategy.
- **Operations Analyst**: Optimizes business operations using data.
- **Quantitative Analyst**: Applies data to financial analysis and risk assessment.

## Advantages of Data Engineering Skills Beyond Data Roles

- **Enhanced Problem-Solving**: Develops structured approaches to solving complex issues.
- **Logical Thinking**: Fosters a logical mindset beneficial for strategic decision-making.
- **Technical Skills**: Provides technical acumen applicable in many modern tech roles.
- **Data Literacy**: Equips with the ability to understand and use data effectively.
- **Project Management**: Aligns with managing projects, resources, and workflows.
- **Effective Communication**: Improves the ability to communicate complex ideas clearly.
- **Adaptability**: Prepares for quick adaptation to industry changes.
- **Automation Knowledge**: Offers insights into streamlining and automating processes.
- **Interdisciplinary Collaboration**: Encourages working across various teams and departments.

Learning data engineering skills can significantly enhance your analytical and technical capabilities, useful in a wide array of professions, not limited to traditional data-centric roles.
Loading

0 comments on commit b9d1612

Please sign in to comment.