江湖数字化平台
登录

Jianghu应用功能点

12147

webRtc

WebRTC是一个免费的开源项目，它使得浏览器和移动应用程序之间可以进行实时的音频、视频和数据通信。它建立在WebRTC API之上，该API是一组标准化通信协议的规范，用于浏览器和其他终端之间的通信。WebRTC使用各种技术，包括Opus和VP8编解码器用于音频和视频，以及DataChannel API用于点对点数据传输。WebRTC的一个关键优势是它能够在不需要第三方插件或软件的情况下实现安全、低延迟的通信。这使得它成为视频会议、在线游戏、文件共享等应用的流行选择。

注意事项

  • 使用了SRS服务端，与普通WebRtc使用不太一样
  • 当web发起请求开麦或播放的时候要new一个webRtc实例，播放的时候，有几个播放流就要有几个webRtc实例。

引入方式

  1. //pubspec.yaml 中声明
  2. dependencies:
  3.   flutter_webrtc:
  4.     path: "../jianghu_packages/flutter_webrtc-0.8.6"
  6. //引入
  7. import 'package:flutter_webrtc/flutter_webrtc.dart' as webrtc;

工作流程

参考：/jianghuAppBrowser/jianghuBrowser/lib/webRtc/SrsSdk.dart

  1. 通过webrtc.createPeerConnection()实例化RTCPeerConnection对象，用于建立实时通信。
  1. loopbackConstraints = <String, dynamic>{
  2.   'optional': [
  3.     {'DtlsSrtpKeyAgreement': true}
  4.   ]
  5. };
  6. webrtc.RTCPeerConnection? _pc = await webrtc.createPeerConnection({'sdpSemantics': "unified-plan"}, loopbackConstraints);
  1. 添加音视频通道配置，包括设置流方向，如RecvOnly(只接受)、SendOnly(只发送)等。
  1. //音频
  2. _pc!.addTransceiver(
  3.   kind: webrtc.RTCRtpMediaType.RTCRtpMediaTypeAudio,
  4.   init: webrtc.RTCRtpTransceiverInit(direction: direction),
  5. );
  6. //视频
  7. _pc!.addTransceiver(
  8.   kind: webrtc.RTCRtpMediaType.RTCRtpMediaTypeVideo,
  9.   init: webrtc.RTCRtpTransceiverInit(direction: direction),
  1. 调用_pc!.createOffer()方法，生成当前设备媒体参数，确定是否可以接收音视频流。
  1. offer = await _pc!.createOffer({
  2.   'mandatory': {'OfferToReceiveAudio': !isPublish, 'OfferToReceiveVideo': !isPublish},
  3. });
  1. 将媒体参数放入系统，设置本地说明。
  1. await _pc!.setLocalDescription(offer!);
  1. 调用getSrsAnswer()方法，将生成的媒体参数发送给服务器，在交换answer时，需要定义解码模式等详细信息。
  1. //调用getSrsAnswer()方法
  2. while (answer == null && !isClosed) {
  3.   EasyLoading.showToast(toast == true ? '通讯异常，正在重连${count > 1 ? '.$count.' : ''}' : '通讯连接中${count > 1 ? '.$count.' : ''}',
  4.       duration: const Duration(seconds: 30), toastPosition: EasyLoadingToastPosition.bottom);
  5.   answer = await getSrsAnswer(webRtcUrl!, offer!, isPublish ? 'publish' : 'play');
  6.   count++;
  7.   await Future.delayed(const Duration(milliseconds: 1500));
  8. }
  10. //getSrsAnswer 片段
  11. //webRtcHostUri是应用域名。如果是推流，type=publish，如果是接收，type=play。
  12. String api = "${Constants.webRtcHostUri}rtc/v1/$type/";
  13. Response<dynamic> result = await DioUtil.dioClient
  14.     .post("${Constants.webRtcHostUri}rtc/v1/$type/", data: {"api": api, 'tid': Constants.random100_200, 'streamurl': webrtcUrl, 'clientip': null, 'sdp': offer.sdp});
  1. 接收到服务器返回的answer以后，调用以下方法，设置远程说明。
  1. _pc!.setRemoteDescription(answer!)
  1. 如果是需要录音，可根据需要获取本地录音流，并将其添加到_pc。在添加流时还可以禁用视频通道。
  1.  publishTask() async {
  2.   GetUserMediaSample userMedia = GetUserMediaSample();
  3.   userMedia.startPublish((webrtc.MediaStream? stream, String? type) async {
  4.     //拿到流
  5.     _localStream = stream;
  6.     //check声音
  7.     webrtc.Helper.setVolume(100, _localStream!.getAudioTracks().first);
  8.     //循环流
  9.     _localStream!.getTracks().forEach((track) {
  10.       //把视频通道禁用
  11.       if (track.kind == 'video') {
  12.         track.enabled = false;
  13.       }
  14.       //把流添加到_pc
  15.       _pc!.addTrack(track, _localStream!);
  16.     });
  17.   }, 'audio');
  18. }
  20. //代码来源：/jianghuAppBrowser/jianghuBrowser/lib/webRtc/GetUserMediaSample.dart
  21. void startPublish(OnStreamOK _onStreamOK, type) async {
  22.     onStreamOK = _onStreamOK;
  23.       //配置流参数，web端和app端配置尽量一样，如果参数不一样，会返回400 日志提示设备配型不对
  24.     final mediaConstraints = <String, dynamic>{
  25.       'audio': true,
  26.       'video': {
  27.         'mandatory': {
  28.           'minWidth': '320', // Provide your own width, height and frame rate here
  29.           'minHeight': '240',
  30.           'minFrameRate': '30',
  31.         },
  32.         'facingMode': 'user',
  33.         'optional': [],
  34.       }
  35.     };
  36.     try {
  37.         //拿到流
  38.       MediaStream stream = await navigator.mediaDevices.getUserMedia(mediaConstraints);
  39.       _localStream = stream;
  40.         //设置流
  41.       _onStreamOK(stream, type);
  42.       //  切换清晰度，保持声轨
  43.       //  可以新建个stream -> 移出旧的track(removeTrack) -> 再添加新的视频的track（addTrack）
  44.     } catch (e) {
  45.       debugPrint(e.toString());
  46.     }
  47.     _inCalling = true;
  48.   }

以上就是WebRTC的基本工作流程。

在具体的应用实现中，我们还需要根据具体的需求，调用不同的方法和参数，例如setVolume()方法、startPublish()方法等，来实现音视频录制、数据通道的建立等功能。另外，我们还可以对WebRTC进行优化，以提高实时音视频通信的质量和性能。

推流

  • web端发起推流请求
  1. //代码来源：jianghuAppBrowser/jianghuBrowser/lib/layout/WebViewHandler.dart
  2. case 'senderPublish':
  3.       PublishInstance().create("${result['actionTime']}", result['url']);
  4.       break;
  • 实例SrsSdk，实例化的时候不需要new
  1. //代码来源：/jianghuAppBrowser/jianghuBrowser/lib/webRtc/PublishInstance.dart
  2. create(String callBackId, String? url) {
  3.     publisher = SrsSdk(publish: true, callBackId: callBackId, url: url);
  4. }

FLV (fijkPlayer)

使用tcp协议，只有播放功能，声音比较稳定。

  1. //拦截web端Jessibuca播放功能，转发到app端处理
  2. //代码来源：/jianghuAppBrowser/jianghuBrowser/assets/jianghuJsBridge.js
  3. function initJessibucaProClass() {
  4.   if(window.JessibucaPro) {
  5.     window.JessibucaPro = function(option) {
  6.       this.play = (flvUrl) => {
  7.         this.flvUrl = flvUrl;
  8.         jianghuBridgePostMessage({ action: 'flvPlay', url: flvUrl}, (action, message) => {});
  9.       };
  10.       this.setVolume = (volume) => {};
  11.       this.destroy = () => {
  12.         jianghuBridgePostMessage({ action: 'flvDestroy', url: this.flvUrl}, (action, message) => {});
  13.       };
  14.     }
  15.   } else {
  16.     setTimeout(() => {
  17.       initJessibucaProClass();
  18.     }, 300)
  19.   }
  20. }
  22. // app端接收flv事件
  23. //代码来源：/jianghuAppBrowser/jianghuBrowser/lib/layout/WebViewHandler.dart 
  24. case 'flvPlay':
  25.   FlvHandler().playFlv(false, result['url']);
  26.   break;
  27. case 'flvDestroy':
  28.   FlvHandler().close();
  30. //处理相关事件，网络变化重连
  31. //参考：/jianghuAppBrowser/jianghuBrowser/lib/webRtc/FlvHandler.dart
  32. void playFlv(bool localMuted, String flvUrl) {
  33.   _flvUrl = flvUrl;
  34.   Map json = {"flvUrl": flvUrl};
  35.   if (flvJson.value == null || !flvJson.value!.containsKey('flvUrl') || flvJson.value!.containsKey('flvUrl') && flvJson.value!['flvUrl'] != flvUrl) {
  36.     flvJson.value = json;
  37.     RenderKey.needFlv.value = true;
  38.   }
  39.   RenderKey.connectivityResult.addListener(onNetworkChange);
  40. }
  41. void close() {
  42.   flvJson.value = null;
  43.   fijkState.value = FijkState.idle;
  44.   RenderKey.needFlv.value = false;
  45.   RenderKey.connectivityResult.removeListener(onNetworkChange);
  46. }
  47. void onNetworkChange() {
  48.   if (flvJson.value == null) return;
  49.   if (_flvUrl == null) return;
  50.   // 之前在播放，现在断开了，重链接网络后，重新播放
  51.   if (RenderKey.connectivityResult.value == ConnectivityResult.wifi || RenderKey.connectivityResult.value == ConnectivityResult.mobile) {
  52.     EasyLoading.showToast('通讯异常，正在重连', duration: const Duration(seconds: 300), toastPosition: EasyLoadingToastPosition.bottom);
  53.     flvJson.value = null;
  54.     fijkState.value = FijkState.idle;
  55.     RenderKey.needFlv.value = false;
  56.     Future.delayed(const Duration(milliseconds: 1500), () {
  57.       playFlv(flvLocalMuted.value, _flvUrl!);
  58.     });
  59.   } else {
  60.     print("当前没网络");
  61.   }
  62. }
  64. /*
  65. * 使用fijkplayer包，实现音视频播放
  66. * 参考：/jianghuAppBrowser/jianghuBrowser/lib/webRtc/IjkPlayerContainer.dart
  67. */
  68. import 'package:fijkplayer/fijkplayer.dart';
  69. Future<void> playFlv() async {
  70.   if (FlvHandler().flvJson.value == null) {
  71.     print('FijkPlayer.release()');
  72.     if(fijkPlayer != null && fijkPlayer!.state == FijkState.started) {
  73.       await fijkPlayer!.stop();
  74.     }
  75.     FlvHandler().fijkState.value = FijkState.idle;
  76.   } else {
  77.     fijkPlayer ??= FijkPlayer();
  78.     if (!fijkPlayerInit) {
  79.       await initIjkPlayer();
  80.     }
  81.     FlvHandler().fijkState.value = FijkState.prepared;
  82.     print('flvJson: ${FlvHandler().flvJson}');
  83.     await fijkPlayer!.reset();
  84.     await fijkPlayer!.setDataSource(FlvHandler().flvJson.value!['flvUrl'], autoPlay: true);
  85.     print("fijkPlayer!.setDataSource");
  86.     EasyLoading.dismiss();
  87.     setState(() { });
  88.   }
  89. }

websocketAudio

使用流程：

  1. //核心代码：`/jh_websocket_audio/android/src/main/kotlin/org/fsll/socket_audio_app/SoundStreamPlugin.kt` 
  3. //拦截web端WSAudio相关的事件，转发到app端处理
  4. //代码来源：/jianghuAppBrowser/jianghuBrowser/assets/jianghuJsBridge.js
  5. function initWsAudio() {
  6.   if(window.WSAudio) {
  7.     window.WSAudio = function(option) {
  8.       this.startTalk = () => {
  9.         jianghuBridgePostMessage({ action: 'wsAudioStartTalk'}, (action, message) => {});
  10.       };
  11.       this.stopTalk = () => {
  12.         jianghuBridgePostMessage({ action: 'wsAudioStopTalk'}, (action, message) => {});
  13.       };
  14.       this.setPlayerMute = () => {
  15.         jianghuBridgePostMessage({ action: 'wsAudioSetPlayerMute'}, (action, message) => {});
  16.       };
  17.       this.setPlayerUnMute = () => {
  18.         jianghuBridgePostMessage({ action: 'wsAudioSetPlayerUnMute'}, (action, message) => {});
  19.       };
  20.       this.getPlayerMap = () => {
  21.         jianghuBridgePostMessage({ action: 'wsAudioGetPlayerMap'}, (action, message) => {});
  22.       };
  23.       this.destroy = () => {
  24.         jianghuBridgePostMessage({ action: 'wsAudioDestroy'}, (action, message) => {});
  25.       };
  26.       console.log('WSAudio init')
  27.       jianghuBridgePostMessage({ action: 'initWsAudio'}, (action, message) => {});
  28.     }
  29.   } else {
  30.     setTimeout(() => {
  31.       initWsAudio();
  32.     }, 300)
  33.   }
  34. }
  36. // app端接收wsAudio事件
  37. //代码来源：/jianghuAppBrowser/jianghuBrowser/lib/layout/WebViewHandler.dart 
  38. case 'initWsAudio': // 进房间触发
  39.   WsAudioHandler().initWsAudio();
  40.   break;
  41. case 'wsAudioStartTalk': // 开卖触发
  42.   WsAudioHandler().wsAudioStartTalk();
  43.   break;
  44. case 'wsAudioStopTalk': // 关麦触发
  45.   WsAudioHandler().wsAudioStopTalk();
  46.   break;
  47. case 'wsAudioSetPlayerMute':
  48.   WsAudioHandler().toggleLocalMuted(true);
  49.   break;
  50. case 'wsAudioSetPlayerUnMute':
  51.   WsAudioHandler().toggleLocalMuted(false);
  52.   break;
  53. case 'wsAudioGetPlayerMap':
  54.   break;
  55. case 'wsAudioDestroy': // 离开房间触发
  56.   WsAudioHandler().wsAudioDestroyAll();
  57.   break;
  59. //初始化wsAudio，并处理相关事件
  60. //参考：/jianghuAppBrowser/jianghuBrowser/lib/wsAudio/WsAudioHandler.dart
  61. //流程：初始化socket，判断网络连接状态 -> 定时任务定期检查socket连接状态 -> 监听网络变化，并重连-> player初始化、播放音频流，initPublish、推流
  62. initWsAudio() {
  63.   // socket检查初始化
  64.   checkInstanceAndSocket();
  65.   timeTaskCheckSocket();
  66.   RenderKey.connectivityResult.addListener(onNetworkChange);
  67.   // player初始化
  68.   jhWebsocketAudioInstance!.initPlay();
  69.   // 发送者初始化
  70.   jhWebsocketAudioInstance!.initPublish();
  71. }
  73. /*
  74. * wsAudio相关事件实现
  75. * 参考：/jianghuAppBrowser/jianghu_packages/jh_websocket_audio/lib/jh_websocket_audio.dart
  76. */
  77. //初始化Socket
  78. initSocket() {
  79.   if(socket != null) socket!.destroy();
  80.   socket = IO.io(liveHost.toString(), {
  81.     'path': liveSocket,
  82.     'auth': requestBody,
  83.     'closeOnBeforeunload': true,
  84.     'transports': [ 'websocket' ],
  85.     'forceNew': true,
  86.     'timeout': 5000,
  87.     'pingInterval': 5000,
  88.     'secure': false // 是否支持SSL/TLS
  89.   });
  90.   // 设置持久化连接
  91.   socket!.on('connect', (_) {
  92.     socket!.emit('setPersistence', true);
  93.   });
  94.   //接收audio事件
  95.   socket!.on("audio", (msg) {
  96.     if (!audioMuted && initializePlayer) {
  97.       _player.writeChunk(msg['data'], msg['id']);
  98.     }
  99.   });
  100.   //audio关闭事件
  101.   socket!.on("audio-break", (msg) {
  102.     print("audio-break $msg");
  103.     _player.removePlayer(msgId: msg['id']);
  104.   });
  105.     
  106. }
  107. // 10毫秒接收一次流，tcp网络有抖动，时间不准
  108. initPlay() async {
  109.   await _player.initialize();
  110.   _playerStatus = _player.status.listen((status) {
  111.     print("_playerStatus$status");
  112.   });
  114.   initializePlayer = true;
  115. }
  116. //10毫秒发送一次流
  117. Future<void> initPublish() async {
  118.   _recorderStatus = _recorder.status.listen((status) {
  119.     isRecording.value = status == SoundStreamStatus.Playing;
  120.   });
  122.   _audioStream = _recorder.audioStream.listen((data) {
  123.     socket!.emit('audio', {
  124.       "sts": DateTime.now().millisecondsSinceEpoch,
  125.       "dts": DateTime.now().millisecondsSinceEpoch,
  126.       "samplerate": "16000",
  127.       "data": data,
  128.     });
  129.   });
  131.   await _recorder.initialize();
  132.   initializeRecord = true;
  133. }

openFileX

打开文件、图片，浏览器等。

  1. //pubspec.yaml 中声明
  2. dependencies:
  3.   open_filex:
  4.     path: "../jianghu_packages/open_file-master"
  6. //引入
  7. import 'package:open_filex/open_filex.dart';
  9. //使用
  10. OpenFilex.openBrowser(result['link']);
  11. OpenFilex.open(file.path);

audioMode

web端发送过来的消息，可选择使用听筒、蓝牙、耳机、外放播放。

  1. //代码来源：/jianghuAppBrowser/jianghuBrowser/lib/layout/WebViewHandler.dart
  2. case 'changeAudioMode':
  3.     AudioMode().changeAudioMode(result);
  4.     break;
  6. //通过_methodChannel给安卓或ios系统发送消息，安卓或ios接收到之后，切换设备。
  7. //代码来源：/jianghuAppBrowser/jianghuBrowser/lib/webRtc/audioMode.dart
  8. changeAudioMode(Map result) async {
  9.   if(audioModeInitStatus != 'success') {
  10.     await initialize();
  11.   }
  12.   if (audioModeInitStatus == 'success') {
  13.     if(audioFocus.value == "AUDIOFOCUS_GAIN") {
  14.       await changeAudioModeChannel(audioMode: result['mode']);
  15.     } else {
  16.       await changeAudioModeChannel(audioMode: result['mode']);
  17.     }
  18.   }
  19. }
  20. Future<dynamic> changeAudioModeChannel({String audioMode = 'speaker'}) async {
  21.   requestAudioMode = audioMode;
  22.   _methodChannel.invokeMethod("changeAudioMode", {
  23.     "audioMode": audioMode,
  24.   });
  25. }
  27. //更换模式
  28. //代码来源：/jianghuAppBrowser/jianghuBrowser/android/app/src/main/kotlin/org/fsll/jianghu_browser/AudioModeManagement.kt
  29. @SuppressLint("MissingPermission")
  30. private fun changeMode() {
  31.   if (audioFocus == AudioManager.AUDIOFOCUS_GAIN || audioFocus == AudioManager.AUDIOFOCUS_GAIN_TRANSIENT || audioFocus == AudioManager.AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK) {
  32.     Log.i(this.javaClass.name, "changeMode")
  33.     when (playMode) {
  34.       //耳机
  35.       AudioMode.Headset -> {
  36.         if (mContext.registerReceiver(null, IntentFilter(Intent.ACTION_HEADSET_PLUG))
  37.             ?.getIntExtra("state", 0) == 1
  38.         ) {
  39.           changeToHeadset()
  40.         } else {
  41.           _audioDeviceEvent?.success("没有插入耳机")
  42.         }
  43.       }
  44.       //外放
  45.       AudioMode.Speaker -> {
  46.         Log.d(this.javaClass.name, "切换到外放")
  47.         changeToSpeaker()
  48.       }
  49.       //听筒
  50.       AudioMode.Receiver -> {
  51.         Log.d(this.javaClass.name, "切换到听筒")
  52.         changeToReceiver()
  53.       }
  54.       //蓝牙
  55.       AudioMode.Bluetooth -> {
  56.         Log.d(this.javaClass.name, "切换到蓝牙")
  57.         // 申请打开蓝牙
  58.         if (hasDeviceBluetooth) {
  59.           changeToBluetooth()
  60.         } else {
  61.           _audioDeviceEvent?.success("没有连接蓝牙设备")
  62.         }
  63.       }
  64.     }
  65.   } else {
  66.     _audioDeviceEvent?.success("没有声音控制权")
  67.   }
  68. }

进入房间先申请设备声音控制权，离开房间释放控制权。

  1. //代码来源：/jianghuAppBrowser/jianghuBrowser/lib/layout/WebViewHandler.dart
  2. case 'enterLive':
  3.   await AudioMode().joinRoom();
  4.   break;
  5. case 'leaveLive':
  6.   await AudioMode().outRoom();
  7.   break;
  9. //代码来源：/jianghuAppBrowser/jianghuBrowser/lib/webRtc/audioMode.dart
  10. Future<dynamic> joinRoom() async {
  11.   if(audioModeInitStatus != 'success') {
  12.     await initialize();
  13.   }
  14.   if(audioModeInitStatus == 'success') {
  15.     _methodChannel.invokeMethod("joinRoom", { });
  16.   }
  17. }
  18. Future<dynamic> outRoom() async {
  19.   if(audioModeInitStatus != 'success') {
  20.     await initialize();
  21.   }
  22.   if(audioModeInitStatus == 'success') {
  23.     _methodChannel.invokeMethod("outRoom", { });
  24.   }
  25. }

musicPlayer

参考：音频播放

网络状况

connectivity插件允许 Flutter 应用程序发现网络连接并相应地配置。它可以区分蜂窝连接和 WiFi 连接。

参考：/jianghuAppBrowser/jianghuBrowser/lib/layout/JianghuConfigHandler.dart

  1. //pubspec.yaml 中声明
  2. dependencies:
  3.   connectivity:
  5. //引入
  6. import 'package:connectivity/connectivity.dart';
  8. //获取当前网络状态，WiFi或者流量
  9. RenderKey.connectivityResult.value = await Connectivity().checkConnectivity();
  11. //监听
  12. Connectivity().onConnectivityChanged.listen((ConnectivityResult event) {
  13.   RenderKey.connectivityResult.value = event;
  14. });