NIO

Java NIO(New IO)是从 Java 1.4 版本开始引入的一个新的 IO API,可以替代标准的 Java IO API。NIO 与原来的 IO 有同样的作用和目的,但是使用方式完全不同,NIO 支持面向缓冲区的、基于通道的 IO 操作。NIO 将以更加高效的方式进行文件的读写操作。

其核心思想是**基于通道(Channel)与缓冲区(Buffer)的非阻塞 IO,并通过 Selector 实现 IO 多路复用**,从而让 Java 程序以更高的效率进行文件与网络 IO。

传统 OIO 是面向流、阻塞式、单向的,一个连接需占用一个线程;NIO 借助缓冲区、非阻塞与选择器,用少量线程即可管理大量连接,解决了高并发下线程资源被大量浪费的痛点。
Java 1.4+ / NIO 标准

1. 概述

在 1.4 版本之前,Java IO 类库是阻塞式 IO;从 1.4 版本开始,引进了新的异步 IO 库,被称为 Java New IO 类库,简称为 Java NIO。

Java NIO 类库的目标是要让 Java 支持非阻塞 IO。基于这个原因,更多的人喜欢称 Java NIO 为非阻塞 IO(Non-Block IO),称“老的”阻塞式 Java IO 为 OIO(Old IO)。总体上说,NIO 弥补了原来面向流的 OIO 同步阻塞的不足,它为标准 Java 代码提供了高速的、面向缓冲区的 IO。

**核心组件** Java NIO 类库包含以下三个核心组件:Channel(通道)、Buffer(缓冲区)、Selector(选择器)。Java NIO 属于第三种模型——IO 多路复用模型,组件提供了统一的应用开发 API,屏蔽了底层操作系统的差异。

1.1 NIO 与 OIO 的区别

在 Java 中,NIO 和 OIO 的区别主要体现在三个方面:

对比维度 OIO(Old IO) NIO(New IO)
编程模型 面向流(Stream Oriented),顺序流式读取,不能随意改变读取指针位置,也不能前后移动流中的数据 面向缓冲区(Buffer Oriented),可与 Buffer 任意位置交互,可随意读取、修改任意位置的数据
操作方式 阻塞:调用 read() 或 write() 时线程被阻塞,直到数据被读取或完全写入,期间不能再做其他事 非阻塞:底层数据已准备好则复制到 Buffer,无数据时当前线程可以去干别的事情,无需阻塞等待
选择器 没有 Selector 概念,不需要用到选择器 基于底层 IO 多路复用技术(Windows 的 select、Linux 的 select/poll/epoll),有 Selector 选择器概念
通道/流方向 Stream 单向:InputStream 只读、OutputStream 只写 Channel 双向:既可读也可写

1.2 Channel(通道)

Channel(通道)国内大多翻译成“通道”,其角色和 OIO 中的 Stream(流)差不多。在 OIO 中,同一个网络连接会关联到两个流:一个输入流(Input Stream),另一个输出流(Output Stream);在 NIO 中,一个网络连接使用一个通道表示,所有的 NIO 的 IO 操作都是通过连接通道完成的。一个通道类似于 OIO 中两个流的结合体,既可以从通道读取数据,也可以向通道写入数据。

特性 Stream(OIO) Channel(NIO)
方向 单向:InputStream 只读、OutputStream 只写 双向:既可用于读操作,也可用于写操作
关联数量 一个连接关联两个独立的流(输入流 + 输出流) 一个连接对应一个通道
数据交互 顺序地从流中读取/写入 与 Buffer 交互,可随机读写

NIO 中的 Channel 的主要实现有:

实现类 用途
FileChannel 用于文件 IO 操作
DatagramChannel 用于 UDP 的 IO 操作
SocketChannel 用于 TCP 的传输操作
ServerSocketChannel 用于 TCP 连接监听操作

1.3 Selector(选择器)与 IO 多路复用

IO 多路复用指的是一个进程/线程可以同时监视多个文件描述符(含 socket 连接),一旦其中的一个或多个文件描述符可读或者可写,该监听进程/线程能够进行 IO 事件的查询。

在 Java 应用层面,如何实现对多个文件描述符的监视?需要用到一个非常重要的 Java NIO 组件——Selector 选择器。Selector 选择器可以理解为一个 IO 事件的监听与查询器,通过选择器,一个线程可以查询多个通道的 IO 事件的就绪状态。

2. ByteBuffer

2.1 基本使用

引入 Netty 依赖(示例中用于后续 ByteBuf 等组件):

<dependency>
    <groupId>io.netty</groupId>
    <artifactId>netty-all</artifactId>
    <version>4.1.110.Final</version>
</dependency>

使用 FileChannel 读取文件并通过 ByteBuffer 打印内容:

public static void main(String[] args){
    //FileChannel
    //1.输入输出流
    try (FileChannel channel = new FileInputStream("data.txt").getChannel()){
        ByteBuffer buffer = ByteBuffer.allocate(512);

        //从channel导入并且向buffer导入
        channel.read(buffer);
        //打印出buffer的内容
        buffer.flip();  //切换到读模式
        ArrayList<Byte> bytes =new ArrayList<Byte>();
        while (buffer.hasRemaining()){
            byte b = buffer.get();
            bytes.add(b);
        }
        System.out.println(bytes);
        //切换到写模式
        buffer.clear();
    }catch (Exception e){
        e.printStackTrace();
    }
}

2.2 ByteBuffer 的结构

ByteBuffer 的三个核心属性:

capacity 容量

position 读写的位置

limit 读写的限制

compact 方法 切换至写模式

通过 allocate 与 allocateDirect 对比堆内存与直接内存:

public static void main(String[] args){
  System.out.println(ByteBuffer.allocate(16).getClass());
  System.out.println(ByteBuffer.allocateDirect(16).getClass());
}

class java.nio.HeapByteBuffer 堆内存 读写效率较低,受 GC 的影响
class java.nio.DirectByteBuffer 直接内存 读写效率较高(少一层数据的拷贝),不受 GC 的影响

2.3 读取方法

buffer.get() 读取

rewind() 从头开始读

mark 做一个标记,记录 position 的位置,reset 是将 position 重置到 mark 的位置

get(i) 按索引查找

2.4 字符串转换

“”.getBytes()

ByteBuffer buffer = ByteBuffer.allocate(16);
ByteBuffer bytes = buffer.put("hello".getBytes());

ByteBuffer.warp()

ByteBuffer buffer1 = ByteBuffer.wrap("hello".getBytes());

将 buffer 转为字符串

ByteBuffer buffer1 = ByteBuffer.wrap("hello".getBytes());
//buffer1.flip();

String decode = StandardCharsets.UTF_8.decode(buffer1).toString();
System.out.println(decode);

2.5 Scattering Reads

分散读取

ByteBuffer b1 = ByteBuffer.allocate(4);
ByteBuffer b2 = ByteBuffer.allocate(4);
ByteBuffer b3 = ByteBuffer.allocate(6);
b1.flip();
b2.flip();
b3.flip();

2.6 处理粘包半包

public static void main(String[] args) {
    ByteBuffer source = ByteBuffer.allocate(32);
    source.put("Hello,world\nI'm Zhangsan\nHo".getBytes());
    split(source);
    source.put("w are you?\n".getBytes());
    split(source);
}

public static void split(ByteBuffer source){
    source.flip();
    for (int i=0;i<source.limit();i++){
        if (source.get(i) == '\n'){
            int len = i + 1 - source.position();
            ByteBuffer target = ByteBuffer.allocate(len);

            for (int j = 0;j<len;j++){
                target.put(source.get());
            }
        }
    }
    source.compact();
}

3. FileChannel

**阻塞模式** FileChannel 只工作在阻塞模式下。

3.1 transferTo

public static void main(String[] args) {
    try (
        FileChannel from = new FileInputStream("data.txt").getChannel();
        FileChannel to = new FileOutputStream("data1.txt").getChannel();
    ){
        from.transferTo(0,4096,to);
    }catch (Exception e){
    e.printStackTrace();
    }
}
**大文件传输** 如果传输数据大于 2G,使用 `from.size()` 获取剩余的大小。

3.2 Path 与 Paths

Path 和 Paths 类 JDK7 引入

3.2.1 walkFileTree

public static void main(String[] args) throws IOException {
        Files.walkFileTree(Path.of("D:\\BaiduNetdiskDownload"),new SimpleFileVisitor<Path>(){
            @Override
            public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
                System.out.println(file);
                return super.visitFile(file, attrs);
            }

            @Override
            public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
                System.out.println(dir);
                return super.preVisitDirectory(dir, attrs);
            }
        });
    }

3.2.2 删除多级目录

public static void main(String[] args) throws IOException {
    Files.walkFileTree(Path.of("D:\\BaiduNetdiskDownload"),new SimpleFileVisitor<Path>(){
        @Override
        public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException {
            Files.delete(dir);
            return super.postVisitDirectory(dir, exc);
        }

        @Override
        public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
            Files.delete(file);
            return super.visitFile(file, attrs);
        }
    });
}

3.2.3 拷贝多级目录

String from = "D:\\BaiduNetdiskDownload";
String to = "D:\\BaiduNetdiskDownload";

Files.walk(Paths.get(from)).forEach(path -> {
    try {
        String targetName = path.toString().replace(from,to);
        if (Files.isDirectory(path)){
            Files.createDirectory(Paths.get(targetName));
        }else {
            Files.copy(path, Paths.get(targetName));
        }
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
});

4. 网络编程

4.1 阻塞模式

public class TestNetServer {

    public static void main(String[] args) throws IOException {
        //创建服务器
        ServerSocketChannel serverSocketChannel = ServerSocketChannel.open();
        ByteBuffer buffer = ByteBuffer.allocate(10);
        //绑定监听端口
        serverSocketChannel.bind(new InetSocketAddress(8080));

        List<SocketChannel> channels = new ArrayList<>();
        while (true) {
            //accept 建立连接,进行通信
            SocketChannel accept = serverSocketChannel.accept(); //阻塞方法
            channels.add(accept);

            for (SocketChannel sc : channels){
                sc.read(buffer);
                buffer.flip();

                while(buffer.hasRemaining()){
                    System.out.println(buffer.get());
                }

                buffer.clear();
            }
        }
    }
}
public class TestNetClient {
    public static void main(String[] args) throws IOException {
        SocketChannel socketChannel = SocketChannel.open();
        socketChannel.connect(new InetSocketAddress("localhost",8080));
        socketChannel.write(Charset.defaultCharset().encode("test"));
        System.out.println("waiting");
    }
}

4.2 非阻塞模式

public static void main(String[] args) throws IOException {
    //创建服务器
    ServerSocketChannel serverSocketChannel = ServerSocketChannel.open();
    ByteBuffer buffer = ByteBuffer.allocate(10);
    //绑定监听端口
    serverSocketChannel.bind(new InetSocketAddress(8080));
    serverSocketChannel.configureBlocking(false); //非阻塞模式
    List<SocketChannel> channels = new ArrayList<>();
    while (true) {
        //accept 建立连接,进行通信
        SocketChannel accept = serverSocketChannel.accept(); //非阻塞

        accept.configureBlocking(false);  //配置为非阻塞
        channels.add(accept); //非阻塞

        for (SocketChannel sc : channels){
            sc.read(buffer);
            buffer.flip();

            while(buffer.hasRemaining()){
                System.out.println(buffer.get());
            }

            buffer.clear();
        }
    }
}

4.3 Selector

事件的类型:

事件 触发时机
accept 会在有链接请求时触发
connect 是客户端,连接建立后触发
read 可读事件
write 可写事件
public static void main(String[] args) throws IOException {
    //1. 创建Selector,管理多个channel
    Selector selector = Selector.open();
    ServerSocketChannel serverSocketChannel = ServerSocketChannel.open();
    serverSocketChannel.configureBlocking(false); //非阻塞模式

    //2. SelectionKey事件发生后,通过它可以知道事件和哪个channel发生了事件
    SelectionKey Skey = serverSocketChannel.register(selector, 0, null);
    //只关注accept事件
    Skey.interestOps(SelectionKey.OP_ACCEPT);
    //绑定监听端口
    serverSocketChannel.bind(new InetSocketAddress(8080));
    while (true) {
        //3. selector方法  没有事件发生会阻塞
        selector.select();
        //4 处理事件 内部包含了所有发生的事件
        Iterator<SelectionKey> iterator = selector.selectedKeys().iterator();

        while (iterator.hasNext()){
            SelectionKey key = iterator.next();

            //区分事件类型
            if (key.isAcceptable()){
                ServerSocketChannel channel = (ServerSocketChannel) key.channel();
                SocketChannel sc = channel.accept();
                sc.configureBlocking(false);
                SelectionKey SSkey = serverSocketChannel.register(selector, 0, null);
                SSkey.interestOps(SelectionKey.OP_READ);
            }else if (key.isReadable()){
                SocketChannel channel = (SocketChannel) key.channel();
                ByteBuffer buffer = ByteBuffer.allocate(16);
                channel.read(buffer);
                buffer.flip();
                while(buffer.hasRemaining()){
                    System.out.println(buffer.get());
                }
            }
        }
    }
}

处理空指针异常问题(同时注意 iterator.remove() 与读取完毕关闭通道):

public static void main(String[] args) throws IOException {
    // 1. 创建Selector,管理多个channel
    Selector selector = Selector.open();
    ServerSocketChannel serverSocketChannel = ServerSocketChannel.open();
    serverSocketChannel.configureBlocking(false); // 非阻塞模式

    // 2. 绑定监听端口
    serverSocketChannel.bind(new InetSocketAddress(8080));

    // 3. 注册通道到selector
    SelectionKey Skey = serverSocketChannel.register(selector, SelectionKey.OP_ACCEPT);

    while (true) {
        // 4. selector方法 没有事件发生会阻塞
        selector.select();
        // 5. 处理事件 内部包含了所有发生的事件
        Iterator<SelectionKey> iterator = selector.selectedKeys().iterator();

        while (iterator.hasNext()) {
            SelectionKey key = iterator.next();
            iterator.remove();
            // 6. 区分事件类型
            if (key.isAcceptable()) {
                ServerSocketChannel channel = (ServerSocketChannel) key.channel();
                SocketChannel sc = channel.accept();
                sc.configureBlocking(false);
                // 7. 注册新连接的SocketChannel
                SelectionKey SSkey = sc.register(selector, SelectionKey.OP_READ);
            } else if (key.isReadable()) {
                SocketChannel channel = (SocketChannel) key.channel();
                ByteBuffer buffer = ByteBuffer.allocate(1024); // 调整缓冲区大小
                int bytesRead = channel.read(buffer);
                if (bytesRead == -1) {
                    channel.close(); // 读取完毕,关闭通道
                    continue;
                }
                buffer.flip();
                while (buffer.hasRemaining()) {
                    System.out.print((char) buffer.get());
                }
                buffer.clear(); // 清空缓冲区,准备下一次读取
            }
        }
    }
}

4.4 处理消息边界

LTV 格式,Type 类型 Length 长度 Value 数据 在类型和长度已知的情况下方便获取大小

http 2.0 是 LTV 格式

4.5 处理写消息过多的情况

public static void main(String[] args) throws IOException {
    ServerSocketChannel ssc = ServerSocketChannel.open();
    ssc.configureBlocking(false);

    Selector selector = Selector.open();

    ssc.register(selector, SelectionKey.OP_ACCEPT);
    ssc.bind(new InetSocketAddress(8080));
    while (true){
        selector.select();
        Iterator<SelectionKey> iter = selector.selectedKeys().iterator();

        while (iter.hasNext()){
            SelectionKey key = iter.next();
            iter.remove();
            if (key.isAcceptable()){
                SocketChannel accept = ssc.accept();
                accept.configureBlocking(false);

                StringBuilder stringBuilder = new StringBuilder();
                for (int i=0;i<1000000;i++){
                    stringBuilder.append("a");
                }
                ByteBuffer buffer = Charset.defaultCharset().encode(stringBuilder.toString());
                while (buffer.hasRemaining()){
                    int write = accept.write(buffer);
                    System.out.println(write);
                }
            }
        }
    }
}

客户端读取:

public static void main(String[] args) throws IOException {
    SocketChannel socketChannel = SocketChannel.open();
    socketChannel.connect(new InetSocketAddress("localhost",8080));

    int count = 0;
    while (true){
        ByteBuffer buffer = ByteBuffer.allocate(1024 * 1024);
        count += socketChannel.read(buffer);
        System.out.println(count);
        buffer.clear();
    }
}

输出

131071
262142
393213
524284
655355
786426
917497
1000000

处理消息过多(关注可写事件,使用 attach 挂载未发送完的 Buffer):

public static void main(String[] args) throws IOException {
    ServerSocketChannel ssc = ServerSocketChannel.open();
    ssc.configureBlocking(false);

    Selector selector = Selector.open();

    ssc.register(selector, SelectionKey.OP_ACCEPT);
    ssc.bind(new InetSocketAddress(8080));
    while (true){
        selector.select();
        Iterator<SelectionKey> iter = selector.selectedKeys().iterator();

        while (iter.hasNext()){
            SelectionKey key = iter.next();
            iter.remove();
            if (key.isAcceptable()){
                SocketChannel accept = ssc.accept();
                accept.configureBlocking(false);
                SelectionKey scKey = accept.register(selector, 0, null);
                scKey.interestOps(SelectionKey.OP_READ);

                ByteBuffer buffer = Charset.defaultCharset().encode("a".repeat(100000000));

                int write = accept.write(buffer);
                System.out.println(write);

                if (buffer.hasRemaining()){
                    //关注可写事件
                    scKey.interestOps(scKey.interestOps() + SelectionKey.OP_WRITE);
                    //挂载
                    scKey.attach(buffer);
                }
            }else if (key.isReadable()) {
                ByteBuffer buffer1 = (ByteBuffer) key.attachment();
                SocketChannel sc = (SocketChannel) key.channel();

                int write = sc.write(buffer1);
                System.out.println(write);

                //清理操作
                if (!buffer1.hasRemaining()){
                    key.attach(null);
                    key.interestOps(key.interestOps() - SelectionKey.OP_WRITE);
                }
            }
        }
    }
}

4.6 IO 模型

IO 模型 说明
阻塞 IO 线程在 IO 期间被阻塞,直到数据就绪
非阻塞 IO 无数据时线程可去做其他事,无需阻塞等待
多路复用 一个线程监视多个文件描述符(selector)
信号驱动 由信号通知 IO 就绪
异步 IO 线程不必等待结果,由操作系统回调返回结果

4.6.1 AIO

异步 IO

在进行读写操作时,线程不必等待结果,而是将来由操作系统来通过回调方式由另外的线程来获取结果

public static void main(String[] args) throws IOException {
    try(AsynchronousFileChannel asynchronousFileChannel = AsynchronousFileChannel.open(
        Paths.get("data.txt"), StandardOpenOption.READ)){
        ByteBuffer buffer = ByteBuffer.allocate(16);

        asynchronousFileChannel.read(buffer, 0, buffer, new CompletionHandler<Integer, ByteBuffer>() {
            @Override
            public void completed(Integer result, ByteBuffer attachment) {
                attachment.flip();
                while (attachment.hasRemaining()){
                    System.out.println((char)attachment.get());
                }
            }

            @Override
            public void failed(Throwable exc, ByteBuffer attachment) {
                exc.printStackTrace();
            }
        });
    }catch (Exception e){
        e.printStackTrace();
    }
    System.in.read();
}

5. Netty

5.1 Netty 入门

netty 是一个异步的、基于事件驱动的网络应用框架,用于开发可维护、高性能的网络服务和客户端

服务器端向客户端发送 Hello:

public static void main(String[] args) {
    //启动器,负责组装netty组件
    new ServerBootstrap().group(new NioEventLoopGroup())
        //选择服务器的ServerSocket的实现
        .channel(NioServerSocketChannel.class)
        .childHandler(new ChannelInitializer<NioSocketChannel>() {
            @Override
            protected void initChannel(NioSocketChannel ch) throws Exception {
                //配置具体handler
                ch.pipeline().addLast(new StringDecoder());  //将buffer转换为字符串
                ch.pipeline().addLast(new ChannelInboundHandlerAdapter(){
                    @Override //读事件
                    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
                        System.out.println(msg);
                    }
                });
            }
        }).bind(8080); //bind为绑定的监听端口
}

客户端:

public static void main(String[] args) throws InterruptedException {
        new Bootstrap()
                .group(new NioEventLoopGroup())
                .channel(NioSocketChannel.class)
                .handler(new ChannelInitializer<NioSocketChannel>() {
                    @Override
                    protected void initChannel(NioSocketChannel ch) throws Exception {
                        ch.pipeline().addLast(new StringEncoder());
                    }
                }).connect("localhost",8080)
                .sync() //阻塞方法,直到连接建立
                .channel() //代表连接对象
                .writeAndFlush("Hello"); //发送数据
    }

5.2 核心组件

5.2.1 EventLoop

事件循环对象,本质是一个单线程执行器(同时维护了一个 Selector),里面有 run 方法处理 Channel 上源源不断地 IO 事件

public static void main(String[] args) {
    EventLoopGroup group = new NioEventLoopGroup(2); //io事件,普通任务,定时任务

    EventLoopGroup group1 = new DefaultEventLoop(); //普通任务、定时任务
    //获取下一个事件循环对象
    //EventLoop next = group.next();
    //执行普通任务
    group.next().submit(()->{
        try{
            Thread.sleep(1000);
        }catch (Exception e){
            e.printStackTrace();
        }
        System.out.println("ok");
    });
    System.out.println("main");
}

执行定时任务:

//执行定时任务
group.next().scheduleAtFixedRate(()->{
  System.out.println("ok");
},0,1, TimeUnit.SECONDS);
System.out.println("main");

EventLoop 分工细化(boss 负责 accept,worker 负责 SocketChannel 的 read 和 write):

new ServerBootstrap()
    // boss负责accept 和 worker负责SocketChannel reade和write
    .group(new NioEventLoopGroup(),new NioEventLoopGroup(2))

进一步细分,将耗时较长的 handler 交给独立的 EventLoopGroup:

//创建一个独立的EventLoopGroup  处理耗时较长的handler
EventLoopGroup eventLoopGroup = new DefaultEventLoopGroup(2);
new ServerBootstrap()
    .group(new NioEventLoopGroup(),new NioEventLoopGroup(2))
    .channel(NioServerSocketChannel.class)
    .childHandler(new ChannelInitializer<SocketChannel>() {
        @Override
        protected void initChannel(SocketChannel ch) throws Exception {
            //将耗时较长的处理交给独立的EventLoopGroup
            ch.pipeline().addLast(eventLoopGroup,"handler1",new ChannelInboundHandlerAdapter(){
                @Override
                public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
                    ByteBuf buf = (ByteBuf) msg;
                    System.out.println(buf.toString(Charset.defaultCharset()));
                    ctx.fireChannelRead(msg); //将消息传给下一个handler
                }
            });
        }
    }).bind(8080);

5.2.2 Channel

channel

  • close() 关闭 channel
  • closeFuture() 用来处理 channel 的关闭
    • sync 方法的作用是同步等待 channel 的关闭
    • addListener 方法是异步等待 channel 的关闭
  • pipline() 方法添加处理器
  • write() 方法写入数据
  • writeAndFlush() 方法将数据写入并且刷出

channelFuture 连接问题:

ChannelFuture future = new Bootstrap()
    .group(new NioEventLoopGroup())
    .channel(NioSocketChannel.class)
    .handler(new ChannelInitializer<NioSocketChannel>() {
        @Override
        protected void initChannel(NioSocketChannel ch) throws Exception {
            ch.pipeline().addLast(new StringEncoder());
        }
        // connect方法是一个异步非阻塞的方法   
        // main发起了调用 真正执行的是nio线程
    }).connect("localhost", 8080);
//无阻塞向下执行
future.channel().writeAndFlush("Hello");; //代表连接对象  & 发送数据

channelFuture 处理结果的方式:

//带有Future Promise的类型都是异步方法配套使用,用来处理结果
ChannelFuture future = new Bootstrap()
    .group(new NioEventLoopGroup())
    .channel(NioSocketChannel.class)
    .handler(new ChannelInitializer<NioSocketChannel>() {
        @Override
        protected void initChannel(NioSocketChannel ch) throws Exception {
            ch.pipeline().addLast(new StringEncoder());
        }
        // connect方法是一个异步非阻塞的方法
        // main发起了调用 真正执行的是nio线程
    }).connect("localhost", 8080);
//方法一:使用sync同步处理结果
/*       future.sync(); //阻塞方法,直到连接建立
        //无阻塞向下执行
        future.channel().writeAndFlush("Hello");; //代表连接对象  & 发送数据*/

// 方法二:使用addListener异步处理结果
future.addListener(new ChannelFutureListener() {
    @Override
    //在 nio线程建立好连接后 执行operationComplete
    public void operationComplete(ChannelFuture channelFuture) throws Exception {
        future.channel().writeAndFlush("Hello");
    }
});

channelFuture 关闭问题:

channelFuture 关闭问题

问题:channel.close 是异步的,因此无法在关闭后执行关闭后的操作,可能 channel 会在一秒钟之后才释放

Channel channel = future.sync().channel();
new Thread(()->{
    Scanner sc = new Scanner(System.in);
    while (true){
        String line = sc.nextLine();
        if (line.equals("q")){
            channel.close();
            break;
        }
        channel.writeAndFlush(line);
    }
},"input").start();

正确处理 channelFuture 关闭:

// 方法一 :获取closeFuture对象 同步处理关闭 异步处理关闭
        ChannelFuture closeFuture = channel.closeFuture();
        System.out.println("等待关闭");
        closeFuture.sync();
        System.out.println("处理关闭后的操作");

        //方法二 : 使用addListener
        closeFuture.addListener(new ChannelFutureListener() {
            @Override
            public void operationComplete(ChannelFuture channelFuture) throws Exception {
                System.out.println("处理关闭后的操作");
            }
        });

当我们输入 q 时发现主线程没有结束,处理该问题:

//带有Future Promise的类型都是异步方法配套使用,用来处理结果
NioEventLoopGroup group = new NioEventLoopGroup();
ChannelFuture future = new Bootstrap()
    .group(group)
    .channel(NioSocketChannel.class)
    .handler(new ChannelInitializer<NioSocketChannel>() {
        @Override
        protected void initChannel(NioSocketChannel ch) throws Exception {
            ch.pipeline().addLast(new LoggingHandler(LogLevel.DEBUG));
            ch.pipeline().addLast(new StringEncoder());
        }
        // connect方法是一个异步非阻塞的方法
        // main发起了调用 真正执行的是nio线程
    }).connect("localhost", 8080);
/*        //方法一:使用sync同步处理结果
       future.sync(); //阻塞方法,直到连接建立
        //无阻塞向下执行
        future.channel().writeAndFlush("Hello");; //代表连接对象  & 发送数据*/

/*// 方法二:使用addListener异步处理结果
        future.addListener(new ChannelFutureListener() {
            @Override
            //在 nio线程建立好连接后 执行operationComplete
            public void operationComplete(ChannelFuture channelFuture) throws Exception {
                future.channel().writeAndFlush("Hello");
            }
        });*/

Channel channel = future.sync().channel();
new Thread(()->{
    Scanner sc = new Scanner(System.in);
    while (true){
        String line = sc.nextLine();
        if (line.equals("q")){
            channel.close();
            break;
        }
        channel.writeAndFlush(line);
    }
},"input").start();

/*// 方法一 :获取closeFuture对象 同步处理关闭 异步处理关闭
        ChannelFuture closeFuture = channel.closeFuture();
        System.out.println("等待关闭");
        closeFuture.sync();
        System.out.println("处理关闭后的操作");
        */
//方法二 : 使用addListener
ChannelFuture closeFuture = channel.closeFuture();

//lamda
closeFuture.addListener((ChannelFutureListener) ChannelFuture-> {
    System.out.println("处理关闭后的操作");
    group.shutdownGracefully();
});
//普通模式
closeFuture.addListener(new ChannelFutureListener() {
    @Override
    public void operationComplete(ChannelFuture channelFuture) throws Exception {
        System.out.println("处理关闭后的操作");
        group.shutdownGracefully();
    }
});

5.2.3 Future & Promise

JDK Future:

public static void main(String[] args) throws ExecutionException, InterruptedException {
    //线程池
    ExecutorService service = Executors.newFixedThreadPool(2);
    //提交任务
    Future<Integer> future = service.submit(new Callable<Integer>() {
        @Override
        public Integer call() throws Exception {
            Thread.sleep(1000);
            return 50;
        }
    });
    //主线程通过future获取结果 get阻塞等待
    System.out.println(future.get());
}

Netty Future:

public static void main(String[] args) throws ExecutionException, InterruptedException {
        NioEventLoopGroup group = new NioEventLoopGroup();

        EventLoop eventLoop = group.next();
        //提交任务
        Future<Integer> future = eventLoop.submit(new Callable<Integer>() {
            @Override
            public Integer call() throws Exception {
                Thread.sleep(1000);
                return 50;
            }
        });
        //获取结果
        System.out.println(future.get());
    }

异步获取结果(非阻塞):

public static void main(String[] args) throws ExecutionException, InterruptedException {
    NioEventLoopGroup group = new NioEventLoopGroup();

    EventLoop eventLoop = group.next();
    //提交任务
    Future<Integer> future = eventLoop.submit(new Callable<Integer>() {
        @Override
        public Integer call() throws Exception {
            Thread.sleep(1000);
            return 50;
        }
    });
    //异步获取结果(非阻塞)
    future.addListener(new GenericFutureListener<Future<? super Integer>>() {
        @Override
        public void operationComplete(Future<? super Integer> future) throws Exception {
            System.out.println(future.getNow());
        }
    });
}

Netty Promise(结果的容器):

public static void main(String[] args) throws ExecutionException, InterruptedException {
    //创建promise对象  promise==>结果的容器
    NioEventLoopGroup group = new NioEventLoopGroup();
    EventLoop eventLoop = group.next();
    DefaultPromise<Integer> promise = new DefaultPromise<>(eventLoop);

    new Thread(()->{
        System.out.println("开始");
        try{
            Thread.sleep(3000);
        }catch (Exception e){
            e.printStackTrace();
        }
        promise.setSuccess(999);
    }).start();
    //同步阻塞
    System.out.println(promise.get());
    System.out.println("结束");
}

传递错误的结果:

new Thread(()->{
            System.out.println("开始");
            try{
                Thread.sleep(3000);
            }catch (Exception e){
                e.printStackTrace();
                promise.setFailure(e);
            }
            promise.setSuccess(999);
        }).start();

5.2.4 Handler & Pipline

ChannelHandler 用来处理 Channel 的各种事件

将消息传递给下一个 handler

ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(new ChannelInboundHandlerAdapter(){
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        System.out.println("1");
        ByteBuf buf = (ByteBuf) msg; 
        String name = buf.toString(Charset.defaultCharset());
        super.channelRead(ctx, name);
    }
});
ch.pipeline().addLast(new ChannelInboundHandlerAdapter(){
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        System.out.println("2");
        super.channelRead(ctx, msg);
    }
});

模拟出站入站:

EmbeddedChannel channel = new EmbeddedChannel();

//模拟入站操作
channel.writeInbound(ByteBufAllocator.DEFAULT.buffer().writeBytes("Hello".getBytes()));
//模拟出站
channel.writeOutbound(ByteBufAllocator.DEFAULT.buffer().writeBytes("hi".getBytes()));

5.2.5 ByteBuf

5.2.5.1 创建
ByteBuf buf = ByteBufAllocator.DEFAULT.buffer();
System.out.println(buf);
String s = "a".repeat(300);
buf.writeBytes(s.getBytes());
System.out.println(buf);

输出结果

PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 256)
PooledUnsafeDirectByteBuf(ridx: 0, widx: 300, cap: 512)
5.2.5.2 直接内存和堆内存

堆内存:

ByteBuf buf1 = ByteBufAllocator.DEFAULT.heapBuffer();

直接内存(默认):

ByteBuf buf2 = ByteBufAllocator.DEFAULT.directBuffer();
5.2.5.3 池化和非池化

可以重用 Buf:

System.out.println(buf1.getClass());

//---》class io.netty.buffer.PooledUnsafeHeapByteBuf
5.2.5.4 读取

buf.readByte() 读取一个字节

合并两个 ByteBuf:

//合并两个byteBuf0
ByteBuf buf = ByteBufAllocator.DEFAULT.buffer(10);
buf.writeBytes(new byte[]{2,3,4,5,6,7});

ByteBuf buf1 = ByteBufAllocator.DEFAULT.buffer(10);
buf1.writeBytes(new byte[]{2,3,4,5,6,7});

ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer(20);
buffer.writeBytes(buf).writeBytes(buf1);

优化:使用 CompositeByteBuf 避免重复的复制:

ByteBuf buf = ByteBufAllocator.DEFAULT.buffer(10);
buf.writeBytes(new byte[]{2,3,4,5,6,7});

CompositeByteBuf buffer = ByteBufAllocator.DEFAULT.compositeBuffer();
buffer.addComponents(true,buf, buf1);

6. Netty 进阶

6.1 粘包和半包问题处理措施

6.1.1 定长解码器

new Bootstrap()
                    .group(group)
                    .channel(NioSocketChannel.class)
                    .handler(new ChannelInitializer<NioSocketChannel>() {
                        @Override
                        protected void initChannel(NioSocketChannel ch) throws Exception {
                            //添加定长解码器
                            ch.pipeline().addLast(new FixedLengthFrameDecoder(10));
                            ch.pipeline().addLast(new LoggingHandler(LogLevel.DEBUG));
                        }
                    }).connect("localhost", 8080);

6.1.2 LineBasedFrameDecoder

new Bootstrap()
                    .group(group)
                    .channel(NioSocketChannel.class)
                    .handler(new ChannelInitializer<NioSocketChannel>() {
                        @Override
                        protected void initChannel(NioSocketChannel ch) throws Exception {
                            //添加定长解码器
                            ch.pipeline().addLast(new LineBasedFrameDecoder(10));
                            ch.pipeline().addLast(new LoggingHandler(LogLevel.DEBUG));
                        }
                    }).connect("localhost", 8080);

6.1.3 LengthFieldBaseFrameDecoder

参数 说明
lengthFieldOffset 长度字段偏移量
lengthFiledLength 长度字段长度
lengthAdjustment 长度字段为基准,还有几个字节是内容
initialBytesToStrip 从头剥离几个字节

6.2 协议的设计和解析

HTTP 协议:

public static void main(String[] args) {
    final Logger logger = LoggerFactory.getLogger(HttpServer.class);
    NioEventLoopGroup worker = new NioEventLoopGroup();
    NioEventLoopGroup boss = new NioEventLoopGroup();

    try{
        ServerBootstrap bootstrap = new ServerBootstrap();
        bootstrap.channel(NioServerSocketChannel.class);
        bootstrap.group(boss,worker);
        bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
            @Override
            protected void initChannel(SocketChannel ch) throws Exception {
                ch.pipeline().addLast(new LoggingHandler(LogLevel.DEBUG));
                ch.pipeline().addLast(new HttpServerCodec());
                ch.pipeline().addLast(new ChannelInboundHandlerAdapter(){
                    @Override
                    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
                        if (msg instanceof HttpRequest){

                        }else if (msg instanceof HttpContent){

                        }
                    }
                });
            }
        });
        ChannelFuture future = bootstrap.bind(8080).sync();
        logger.info("Server started at http://127.0.0.1:{}", 8080);
        future.channel().closeFuture().sync();
    }catch (Exception e){
        e.printStackTrace();
    }finally {
        worker.shutdownGracefully();
        boss.shutdownGracefully();
    }
}

根据消息的类型区分(SimpleChannelInboundHandler):

ServerBootstrap bootstrap = new ServerBootstrap();
            bootstrap.channel(NioServerSocketChannel.class);
            bootstrap.group(boss,worker);
            bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
                @Override
                protected void initChannel(SocketChannel ch) throws Exception {
                    ch.pipeline().addLast(new LoggingHandler(LogLevel.DEBUG));
                    ch.pipeline().addLast(new HttpServerCodec());
                    ch.pipeline().addLast(new SimpleChannelInboundHandler<HttpRequest>() {
                        @Override
                        protected void channelRead0(ChannelHandlerContext ctx, HttpRequest msg) throws Exception {
                            //获取请求
                            logger.info(msg.uri());
                            //返回响应
                            DefaultFullHttpResponse response = new DefaultFullHttpResponse(msg.protocolVersion(), HttpResponseStatus.OK);
                            response.content().writeBytes("<h1>test<h1/>".getBytes());

                            //写回响应
                            ctx.writeAndFlush(response);
                        }
                    });
                }
            });

输出结果 [nioEventLoopGroup-2-1] INFO com.sun.net.httpserver.HttpServer - /index.html

response.headers().setInt(HttpHeaderNames.CONTENT_LENGTH,128);

6.3 自定义协议

字段 说明
魔数 第一时间判定是否是有效的数据包
版本号 可以支持协议的升级
序列化算法 消息正文
指令类型 登录、注册、群聊、私聊
请求序号 为了双工通信、提供异步能力
正文长度 消息正文长度
消息正文 实际数据

自定义协议编码示例:

public class Decoder extends ByteToMessageCodec<Message> {
    @Override
    protected void encode(ChannelHandlerContext ctx, Message msg, ByteBuf out) throws Exception {
        //魔数
        out.writeBytes(new byte[]{1,2,3,4});
        //字节版本
        out.writeByte(1);
        //字节的序列化方式 0jdk 1json
        out.writeByte(0);
        // 字节的指令类型
        out.writeByte(2);
        //序列号
        out.writeByte(5);
        //无意义 用来对齐
        out.writeByte(0xff);
        
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        ObjectOutputStream obs = new ObjectOutputStream(bos);
        obs.writeObject(msg);
        byte[] byteArray = bos.toByteArray();
        //正文长度
        out.writeByte(byteArray.length);
        //正文
        out.writeBytes(byteArray);
    }

    @Override
    protected void decode(ChannelHandlerContext channelHandlerContext, ByteBuf byteBuf, List<Object> list) throws Exception {

    }
}
**定义解码** 在 `decode` 方法中按固定顺序读取魔数、版本、序列化方式、指令类型、序列号、正文长度后,再读取正文并反序列化为 `Message` 对象加入结果列表。

完整的解码实现:

public class Decoder extends ByteToMessageCodec<Message> {
    final Logger logger = LoggerFactory.getLogger(Decoder.class);
    @Override
    protected void encode(ChannelHandlerContext ctx, Message msg, ByteBuf out) throws Exception {
        //魔数
        out.writeBytes(new byte[]{1,2,3,4});
        //字节版本
        out.writeByte(1);
        //字节的序列化方式 0jdk 1json
        out.writeByte(0);
        // 字节的指令类型
        out.writeByte(2);
        //序列号
        out.writeByte(5);
        //无意义 用来对齐
        out.writeByte(0xff);
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        ObjectOutputStream obs = new ObjectOutputStream(bos);
        obs.writeObject(msg);
        byte[] byteArray = bos.toByteArray();
        //正文长度
        out.writeByte(byteArray.length);
        //正文
        out.writeBytes(byteArray);
    }

    @Override
    protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> list) throws Exception {
        int magicMsg = in.readInt();
        byte version = in.readByte();
        byte serializeType = in.readByte();

        int sequenceId = in.readInt();
        in.readByte();
        int length = in.readInt();
        byte[] bytes = new byte[length];
        in.writeBytes(bytes,0,length);
        //假设为jdk
        if (serializeType == 0){
            ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
            ObjectInputStream ois = new ObjectInputStream(bis);
            Message msg = (Message)ois.readObject();
            logger.info(msg.toString());
            list.add(msg);
        }
    }
}

7. 聊天业务

聊天服务器示例,使用 LengthFieldBasedFrameDecoder 处理消息边界:

package richuff.top;

import io.netty.bootstrap.ServerBootstrap;
import io.netty.channel.ChannelInitializer;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.SocketChannel;
import io.netty.channel.socket.nio.NioServerSocketChannel;
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class ChatServer{
    static Logger logger =  LoggerFactory.getLogger(ChatServer.class);
    public static void main(String[] args) {
        NioEventLoopGroup boss = new NioEventLoopGroup();
        NioEventLoopGroup worker = new NioEventLoopGroup();
        try{
            ServerBootstrap bootstrap = new ServerBootstrap();
            bootstrap.channel(NioServerSocketChannel.class);
            bootstrap.group(boss,worker);
            bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
                @Override
                protected void initChannel(SocketChannel ch) throws Exception {
                    ch.pipeline().addLast(new LengthFieldBasedFrameDecoder(1024,0,0,0,0));

                }
            });
            bootstrap.bind(8080).sync().channel();
        }catch (InterruptedException e){
            logger.info(String.valueOf(e));
        }finally {
            boss.shutdownGracefully();
            worker.shutdownGracefully();
        }
    }
}

一句话总结:Java NIO 以 Channel、Buffer、Selector 三大核心组件实现了面向缓冲区、非阻塞的 IO 多路复用,并在此之上借助 Netty 提供了事件驱动、高性能的网络编程与自定义协议能力。