网络课程笔记

00:00 Hello and welcome to this lecture, VPC

00:02 traffic monitoring with VPC flow logs.

00:04 Now, in this section introduction lecture,

00:07 we talked about various tools which you can use for traffic monitoring,

00:11 troubleshooting and analysis.

00:13 And out of those VPC flow logs is one of such IAMportant feature.

00:17 So let's talk about what is VPC

00:19 flow logs and then how to use it?

00:22 So in a sIAMple term,

00:23 VPC flow logs allow you to capture the information about

00:26 the IP traffic that is going in and out of your

00:30 ENIs

00:30 ENIs

00:31 are elastic network interfaces.

00:33 And these VPC flow

00:34 logs you can enable at the VPC

00:36 level at

00:37 the Subnet

00:38 level or individual

00:39 ENI level,

00:41 which means if you enable the flow logs at the VPC

00:44 level,

00:45 then the flow logs will be enabled for all the

00:48 ENIs in that VPC.

00:50 Now, what's the purpose of VPC

00:52 flow

00:52 logs? So typically it is used to monitor and troubleshoot any connectivity issues.

00:57 For example, you are not able to reach to particular EC2 instance,

01:01 which means all your connections request are rejected.

01:05 Now, in that case, you can go and check the VPC

01:07 flow logs and see which ENI

01:10 is denying that traffic.

01:12 And then of course,

01:13 you can check the security group roles or

01:15 NACL roles and fix that connectivity problem.

01:19 Another use case is to see if any malicious

01:22 IPs are trying to access your IP addresses.

01:26 And in that case,

01:27 you can also perform analysis like this is my ENI or this is my IP address

01:32 and show me how many requests has hit this IP address.

01:37 And if you know the malicious IPs,

01:39 then you can also filter those results using that malicious IP as a source IP.

01:44 So a lot of such analysis you can do once you capture this VPC

01:48 flow logs.

01:49 And later in this lecture, we will see some of these scenarios.

02:11 Now when we enable the VPC

02:12 flow logs, they are not only enabled for the EC2 ENIs,

02:16 but they are also enabled for

02:20 the

02:20 ENI

02:20 which are created by

02:22 AWS

02:22 services.

02:23 For example, you know that when you use AWS

02:26 services like elastic Load-Balancer or RDS

02:29 databases or

02:31 Amazon workspaces or NAT gateways,

02:33 they also create the ENI

02:35 into your VPC.

02:36 So flow

02:37 log will also capture the logs from these

02:41 ENIs.

02:42 And finally,

02:43 one IAMportant thing to know is that enabling the flow logs doesn't IAMpact your

02:48 ENI's network performance. So do not worry about that.

02:52 OK. With that, let's see how to publish the VPC flow logs. So if you have VPC

02:57 and multiple subnets and then multiple

03:00 EC2 machines,

03:01 of course,

03:01 EC2 will have the ENIs and there could be

03:04 other networking components like load balancers or NAT gateways,

03:08 which also has ENIs inside your VPC

04:23 VPC flow logs can then be sent to

04:26 different systems and can be analyzed in different ways.

04:30 Now, in this lecture,

04:31 we will particularly focus on delivering the flow logs to the cloudwatch

04:36 and then we will analyze that using the Cloudwatch logs insights.

04:41 OK. So when you enable the flow logs, then they have a particular format.

04:46 Now that's called a default format of the VPC

04:48 flow logs.

04:49 And that format looks something like this.

04:52 And we will shortly talk about this format.

04:55 But the default log format for the VPC

04:57 flow log is version 2.

05:00 But since then,

05:01 AWS has launched new versions of VPC flow logs.

05:04 So currently up to version 5 are supported

05:08 and accordingly VPC flow logs can have different fields

05:12 depending on which version of VPC

05:14 flow log you are using.

05:15 So this format, what you see here is the version 2 format.

05:20 Now this is a sample flow log.

05:22 And if you see here there is a version number 2, then this is AWS account ID.

05:27 This is ENI

05:29 now there is a source address from where the packet was

05:32 sent and then the destination address where the packet is going

05:36 sIAMilarly the source port and the destination port

05:40 and then network protocol.

05:42 So you know that TCP is represented as number 6,

05:45 then how many packets were sent how many bytes were sent

05:48 the start tIAMe, the end tIAMe.

05:51 And the IAMportant thing here is this "action".

05:54 So action is accept that means this packet was accepted by the ENI

06:00 and finally the log status, which is OK,

06:02 which means these logs have been captured in the destination.

06:07 So this is a default log format and we will also see this in console.

06:11 Now VPC

06:12 flow log format can also be customized.

06:15 And as I said, depending on which version of VPC flow log you are using,

06:18 you can have additional fields.

06:21 So there are all these additional fields which you can use

06:24 with custom VPC flow log format

06:27 and there are more fields.

06:28 But I think these are the fields which are interesting to you.

06:32 And in the next slide, I will explain some of these fields.

07:03 So

07:04 the record will look something like this.

07:05 Now here we are using the custom flow logs format.

07:09 So this is not a default VPC flow logs format,

07:12 but we have added additional fields.

07:14 So in here, if you see the version is number 5,

07:17 because we are using the maxIAMum fields here.

07:44 Now additional field also supports the type of the packet. So it's IPv4

07:51 and then

07:52 number of packets 14 and number of byte is this,

07:56 then there will be AWS account id.

07:58 And additionally, you can also specify, I want to have the VPC ID as well as the Subnet

08:04 id.

08:05 And along with that you can also include the instance ID.

08:09 And then of course, you can have the ENI ID itself.

08:13 And additionally, you can have AWS Region, AWS availability zone.

08:17 And additionally, there are fields like sublocation type and sublocation ID.

08:22 Now this represents the Amazon wavelength zone or outpost or local zone.

08:28 So if your ENI resides in any of these AWS locations,

08:32 then these fields will be populated,

08:36 then the action which is like whether this packet was accepted or rejected.

08:41 So here it is ACCEPT

08:43 and then there are additional TCP flags.

08:46 Now this is all about the TCP handshake and then you can refer

08:50 AWS documentation what 19 means.

08:54 Now additionally, there are these two fields which are packet source addr

08:59 and packet destination

09:01 addr.

09:01 Now these are

09:03 used to actually know the original packet IP address.

09:07 Now this is because if you are having intermediate devices in the flow, for example,

09:12 the NAT gateways,

09:13 then you know that the packet source address changes.

09:16 But if you want to know the original IP from where the packet was originated,

09:20 then you can use this packet

09:22 source addr

09:24 and same applies to the packet destination addr

09:27 as well.

09:45 Further, there is additional field called traffic path, which is used for the

09:50 egress traffic. And in this case, it is an

09:52 ingress traffic and that's where you see this field is not populated.

09:56 And finally,

09:57 other IAMportant field is the traffic flow direction which tells you whether it was

10:02 ingress traffic or

10:04 egress traffic. Now, in this case, it's an

10:07 ingress traffic

10:08 and finally, the log status, which is OK.

10:11 So you can very well have your own custom log format

10:15 and you can include as many fields as you want.

10:19 So this is just an example of one of such record.

10:22 Okay, So I hope that is clear.

10:24 Now with that, let me quickly show you how to capture the VPC

10:28 flow logs and how to see that in Cloudwatch logs.

10:32 So for capturing the VPC flow logs,

10:34 there are some prerequisites because you are going to

10:37 send these logs to the Cloudwatch log groups.

10:40 So first you need to create corresponding Cloudwatch log groups

10:43 and then you need to create an IAM role for VPC flow logs

10:47 service to send it to the Cloudwatch.

10:49 And I have included this IAM role policy which you need to attach.

10:54 Now let me go to

10:55 AWS console and show you these things,

10:58 okay. So I'm into my cloudwatch console in Mumbai Region and in the logs, I already

11:05 have this logs group and I had just created this log group, right?

11:10 And as I said, you also need to have

11:12 IAM role. So I had created this VPC flow logs role. And if you see the policy,

11:18 it allows to write to the Cloudwatch logs and

11:21 the trust policy is for VPC flow log

11:25 service.

11:26 So these are the prerequisites before you enable the VPC flow logs.

11:30 And of course, then you have to just go to your VPC.

11:33 Now,

11:33 what I have already done is that I have created a VPC with one

11:36 public Subnet and inside that I have launched a single EC2 instance.

11:41 And then I just went here and I said,

11:45 create the VPC flow logs and most of the settings as default

11:49 just that you have to select the destination log group and the corresponding

11:53 IAM role.

11:54 And that's where I have the VPC flow log created. Now let me show you the

11:59 EC2 machine

12:01 which I had launched and this machine has a web server

12:06 and it has a public IP address.

12:08 So in order to generate some traffic, what I can do is I can just go here and

12:13 just hit this web server over port 80.

12:17 And then this page appears that means traffic is

12:20 going to my EC2 instance over port 80.

12:23 So that's the setup I already did.

12:26 And now we want to analyze these logs using cloudwatch logs insights.

12:30 So for that, let's go here and let's go to the logs insights.

12:34 Now logs insights provides you the custom

12:37 query language which is native to Cloudwatch.

12:40 And if you know about the log monitoring tool like Splunk,

12:43 it also uses similar language,

12:46 but this is very powerful in the sense

12:48 that it automatically detects your log record format.

12:51 And then based on some deli-

12:53 meter,

12:53 it extracts automatically the fields from the logs.

12:56 And then you can do a lot of operations with this field,

12:59 for example aggregation, sum or filtering all

13:02 that is possible with this query language.

13:05 And if you need some help, then you can just go here

13:08 and then you can look at some sample queries, for example VPC

13:11 flow

13:11 log queries. Likewise, right.

13:13 OK. So

13:14 the first thing that you need to do is to select

13:16 the log group on which you want to run these queries.

13:19 So I'm selecting my log group

13:21 and now this is a default query. Now, out of this, I don't want few things.

13:25 For example, I'm not interested in log

13:27 streams.

13:28 So for VPC flow

13:29 logs for every ENI,

13:31 the cloudwatch will create a separate log stream.

13:34 So if you're interested,

13:35 you can query a particular log stream or it will query all the log streams.

13:40 That means all the ENIs

13:42 so let's just remove this and run this query.

13:46 OK. So if you see it shows the entire record.

13:50 So it is similar to what we had seen earlier,

13:52 the default format with 2 as the flow log version, the

13:57 AWS account ID, the ENI ID

14:00 and the source IP, the destination IP. And this IP is EC2 private IP

14:06 and this is my public IP.

14:08 So if you go to what is my IP address, it will show you that as IP address

14:13 and that's where in the flow log you will see this IP.

14:16 Now you can also see other IPs and probably this is

14:19 someone else is trying to connect to my EC2 instance.

14:21 And that's where you see that.

14:23 So this record is exactly similar to what we already saw

14:26 and it has all these fields that we already discussed about.

14:30 So likewise, you can run many other queries

14:33 and I leave it up to you to explore and run more queries.

14:37 And for your reference,

14:38 I have provided some sample queries as a text document along with this lecture.

14:42 OK. So going back to our discussion. So we already saw how to create VPC flow logs.

14:48 And these are sample queries that you can use.

14:51 Now,

14:51 one more thing like sometimes it happens that your security group or

14:55 Network-ACL blocks some traffic or say they do not allow the traffic.

14:59 And in that case,

15:00 you can use this action field which is like ACCEPT or REJECT and you can analyze

15:05 your traffic flow and understand that whether this

15:08 traffic was blocked because there was no security group

15:11 or because there was no Network-ACL rule. So let's take an example.

15:16 So for the incoming request, as you know, it first hits the NACL

15:20 and from there it goes to the EC2.

15:23 So one scenario is if your inbound traffic is rejected,

15:27 in that case,

15:28 the reason could be either SG or NACL,

15:31 right?

15:32 But if your inbound traffic was accepted, but the outbound was rejected.

15:37 Then the problem is with

15:38 NACL,

15:39 that's because security groups are stateful,

15:41 which means they automatically allow the return traffic.

15:45 But NACL are stateless.

15:47 And that's where if the inbound packet was accepted, but the outbound was rejected,

15:53 then it's because of the NACL

15:55 rules.

15:56 And also the other way around for the outgoing request.

15:59 If the outbound traffic is rejected, then the reason could be either SG or NACL.

16:05 But if the outbound traffic was accepted,

16:09 but the returning inbound traffic was rejected, then again, the problem is with NACL.

16:15 So I have included this scenario because we

16:17 have seen question in the exam around this.

16:19 Also, it is worth if we just go to

16:22 AWS console and try to simulate this scenario.

16:25 Now, here we will create a custom VPC

16:27 flow logs and then

16:29 we will see what impact the

16:31 NACL rules have or the security group rule have.

16:34 And for that, we will use this query which just filters the traffic for

16:39 myIP as a source or destination address.

16:43 Okay. So first thing let's create the new log group

16:46 and let's call it VPC flow logs,

16:50 custom format

16:52 and rest of the settings as default.

16:56 And now let's go to the VPC

16:57 flow logs actions and say create a flow log.

17:02 And let's call it VPC flow logs custom format.

17:08 And we want all three type of the traffic, whether accepted, rejected or all

17:13 and the aggregation level as one minute.

17:16 Now, depending on this aggregation interval,

17:19 you might have more records or less records.

17:21 So as granular as you go, you will have more records in your VPC flow logs.

17:27 Now, as a destination, we are sending it to the Cloudwatch logs.

17:30 And here we are going to select our custom format log group

17:34 and the same flow log

17:36 IAM role that we had created earlier.

17:39 Now, the format is custom. So let's see which fields are of our interest.

17:43 So we need account ID, we need "action" AZ

17:46 ID not interested

17:48 bytes okay, source and destination address,

17:51 start and end time. Flow direction is important where it's ingress or

17:56 egress

17:56 instance id, interface id, log status

17:59 number of packets.

18:01 Now, here we are not dealing with

18:03 AWS

18:03 services. So it doesn't make sense to have this. And also we don't have intermediate

18:08 gateways. So I'm not selecting this packet source or destination addr

18:14 protocol is important, Region we already know. But let's select that

18:18 source address, source port, start time,

18:21 sub location IDs we don't want because we know that we are operating in Region. Subnet

18:26 ID makes sense. Traffic path is important.

18:29 Then type is IPv4 or IPv6, then version ID and the VPC ID.

18:35 OK. So all these are our fields and that's it. Let's create the flow logs.

18:42 Okay. So flow

18:43 logs has been created and we have to wait maximum

18:46 up to 10 minutes for these logs to be delivered.

18:49 Now until then we can just hit some more traffic from my IP address to this web server.

18:55 And that's this EC2 instance, right?

18:58 So let's do that.

19:00 And let's go to the Cloudwatch logs again

19:03 and let's go to the Cloudwatch logs insights. And here this time,

19:09 let's select our new log group. Now, it doesn't show up here.

19:12 So sometimes you have to refresh.

19:15 So let's see.

19:17 Let's go to the Cloudwatch logs insights. And

19:23 here, let's see if we have this custom format. Yes, we have it

19:27 and

19:28 let's try to run our query.

19:33 So mostly it won't have any data as of now,

19:35 which is expected because it takes up to 10 minutes for these logs to be delivered.

19:40 So I'm pausing this video for almost 5 to 10 minutes and I'll come back here.

19:46 OK?

19:46 So it's already been five minutes and I have

19:49 just changed the query a little where I have included

19:52 the field names because otherwise there will be too

19:54 many fields and they could be in different order.

19:57 So let's run this query

19:59 and you can see that

20:01 there is the result.

20:03 So it tells me what's the interface ID, source IP address, destination address,

20:07 which is the EC2 instance IP address,

20:10 then source port, destination port, flow direction, action and log status.

20:15 Now, what we are really interested in here is

20:19 where the source IP or the destination IP is my own IP address.

20:23 So let's say filter

20:26 source addr

20:29 equal to

20:31 my IP address, which I can get from here

20:37 or we can say

20:39 dstaddr

20:41 equal to

20:43 my IP address.

20:51 And let's run this query again.

20:55 So it will show you that I'm trying to hit to port 80

20:59 then it responds with the

21:00 egress traffic on ephemeral port and both

21:04 ingress and

21:04 egress traffic is accepted because I am able to hit this web server right

21:09 now. Let's quickly do one thing.

21:11 Let's go to the Network-Access-Control-List and then try to block the traffic

21:17 through the NACL

21:18 rules. So if you go to the public Subnet

21:21 in which I have EC2 instance, and if you go to the Network-Access-Control-List,

21:25 which is here,

21:26 let's

21:27 remove the outbound rule and

21:30 let's remove this rule. Now what this means. Now

21:34 the inbound traffic is allowed but the outbound traffic is not allowed,

21:39 that means the return traffic.

21:41 And hence now, if I just open a new window,

21:46 then

21:47 it shouldn't really work. Now you see this coming up because it had cashed the page.

21:52 But if I do this again,

21:55 then you can see that it is not able to connect.

21:58 So

21:59 with that, if we run the cloudwatch logs

22:01 query again,

22:02 then it should show us both the accept and the reject traffic.

22:07 But as you know, it will take some time.

22:09 So I'm again pausing this video for maybe five minutes and then I'll come back.

22:14 OK. It's been almost three minutes. So let's try to run this again

22:19 and OK, there are many records

22:22 and OK, if you see this. So

22:25 this entry is like I'm hitting on this EC2 instance on port 80.

22:30 And then the next entry should be the response from the EC2 to my IP address.

22:35 And this is a destination ephemeral port from where

22:38 I had requested this connection.

22:41 And it

22:42 says

22:42 the

22:43 egress is rejected

22:44 because as you know, the Network-ACL is blocking that traffic.

22:49 So this is what we wanted to simulate.

22:53 Okay. So that's what we wanted to see.

22:55 I haven't really done the security group rule thing.

22:58 But if you are interested, you can try that in your environment

23:02 and just advise here

23:04 if you are trying this exercise, then also, you know,

23:07 disable the VPC flow logs because otherwise

23:10 there will be a charge for Cloudwatch logs group

23:14 and I will also do that after this lecture,

23:17 but just be aware of that.

23:19 OK? So just few more things about the flow logs limitation

23:23 that VPC flow logs currently do not record the traffic for Amazon

23:28 DNS server which runs on VPC base address + 2 IP address.

23:33 And you already know about Amazon DNS server. So these logs are not captured

23:37 any logs,

23:38 which is about the communication with EC2 metadata

23:42 service or Time sync service is also not captured.

23:45 The

23:46 DHCP

23:47 traffic is not captured

23:49 and same with windows license activation server or mirror traffic.

23:53 Now, in the next lecture, we are going to see how to mirror the traffic.

23:56 Now, apart from the native tools like VPC flow

23:59 log, you can also use traditional networking tools to capture the packets.

24:03 For example, you can use wireshark or

24:05 tcpdump

24:06 or you can use other tools like traceroute which

24:09 shows you hop by hop the connection to the destination.

24:13 You can use

24:13 telnet or nslookup kind of tools for DNS resolution.

24:18 And simplest of all these to check the

24:20 connectivity is the PING protocol which gives you

24:23 the RTT that is round trip time and it uses the ICMP protocol for that.

24:30 Now ICMP is not a TCP or a UDP protocol. It is internet control message protocol.

24:36 And for allowing the ping in

24:38 your security group and Network-Access-Control-List,

24:41 you have to explicitly open the ports for all ICMP-IPv4 or IPv6.

24:48 So just remember that

24:49 because sometimes you try to ping to the EC2

24:51 instance and despite everything is all well setup,

24:55 but still you are not able to ping because security group or

24:58 network ACL doesn't have ICMP rule added in that.

25:03 OK. With that, we'll stop here.

25:05 So I hope you now understand what is VPC flow logs and

25:08 how to enable VPC flow logs to capture the packet information.